Development of a Console LLM Chat in Python with Ollama and LiteLLM

Building an Interactive Console LLM Chat in Python with Ollama and LiteLLM

In the context of growing interest in local large language models (LLMs) and their integration into custom applications, it's crucial not only to run a model but also to create a fully functional interactive interface. This article, a continuation of our series, focuses on transforming a basic script for a single LLM query into a functional console chat in Python. We'll cover key steps for structuring code, implementing system prompts, adding interactivity, and handling errors, using the powerful combination of Ollama and LiteLLM. The goal is to demonstrate how to evolve from a simple command execution to building a robust and manageable application capable of real-time user interaction, and to lay the groundwork for further functional expansion, including dialogue context management.

From a One-Shot Script to an Interactive Application

The previous part of our guide demonstrated basic integration of a local LLM, hosted via Ollama, with Python code using the LiteLLM library. The result was a script capable of sending a single predefined query to the model and displaying the response, after which the program terminated. While sufficient for an initial introduction to the technology, this approach is impractical for any real-world application. A true chatbot needs to provide continuous interaction: accepting user input, processing it, receiving a response from the model, and then awaiting the next question until the user decides to end the session. This fundamental transformation is precisely what we will implement, turning a static script into a dynamic console chat.

Key enhancements to be introduced in this part include:

• Extracting the LLM query logic into a separate, reusable function.
• Utilizing a system prompt to define the assistant's role and behavior.
• Organizing a conversation loop, allowing the user to ask multiple questions consecutively.
• Implementing response time measurement to evaluate performance.
• Basic exception handling to improve application robustness.

Understanding the message format that LLM APIs work with is fundamental for effective interaction. Models don't operate on a single line of text; they process a list of messages, each with a specific role:

• system: This is an instruction for the model, defining its personality, response style, limitations, or important contextual data. The user doesn't directly see this message; it's generated at the code level.
• user: The message or question originating from the user.
• assistant: The response generated by the model itself. In future iterations, previous model responses will be added here to maintain context.

Architecture of a Console LLM Chat in Python

Now, let's move on to the implementation. We will modify the main.py file to encapsulate the LLM interaction logic and organize the user input loop. Below is the updated code that you can use to replace the contents of your main.py.

# -*- coding: utf-8 -*-
import time
from typing import Optional
from litellm import completion
 
MODEL = "ollama_chat/qwen2.5:3b"
API_BASE = "http://localhost:11434"
SYSTEM_PROMPT = "You are a helpful assistant. Answer concisely and to the point."
 
 
def send_request_to_llm(user_message: str) -> Optional[str]:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ]
    try:
        start_time = time.time()
        response = completion(
            model=MODEL,
            messages=messages,
            api_base=API_BASE,
            request_timeout=120,
        )
        duration = time.time() - start_time
        print(f"\n⏱ Generation time: {duration:.2f} sec")
        return response.choices[0].message.content
    except Exception as e:
        print(f"\n❌ Error during request: {e}")
        return None
 
 
def main() -> None:
    print("🤖 Local AI assistant started.")
    print("Enter your question or 'exit' to quit.\n")
 
    while True:
        user_input = input("👤 You: ").strip()
 
        if user_input.lower() in ("exit", "exit", "quit"):
            print("Goodbye!")
            break
 
        if not user_input:
            print("⚠️ Please enter a question.")
            continue
 
        print("\n🤔 Model is thinking...")
        answer = send_request_to_llm(user_input)
 
        if answer is not None:
            print(f"\n🦙 AI: {answer}\n")
        else:
            print("\n⚠️ Failed to get a response. Check if Ollama is running.\n")
 
 
if __name__ == "__main__":
    main()

Analyzing Key Code Components

Let's take a closer look at how each block of this script works.

1. Configuration Constants:

At the beginning of the file, constants are defined: MODEL, API_BASE, and SYSTEM_PROMPT. This approach allows for easy modification of configuration parameters, such as using a different LLM model or adjusting the assistant's behavior, without delving into the core logic of the main code. Here, SYSTEM_PROMPT plays a crucial role, setting initial instructions for the model that influence the overall tone and style of its responses.

2. The send_request_to_llm Function:

This function (def send_request_to_llm(user_message: str) -> Optional[str]:) is the central element for interacting with the LLM. It takes a user message, constructs a complete list of messages (including SYSTEM_PROMPT and user_message), and sends it via litellm.completion to the local Ollama model.

• Encapsulation: The logic for sending a request and receiving a response is fully encapsulated, making the main main loop cleaner and more understandable.
• Time Measurement: The built-in time measurement (time.time()) allows for evaluating model performance, showing how many seconds it took to generate a response. This is especially useful when experimenting with different models or parameters.
• Error Handling: The try...except block provides basic fault tolerance. If an error occurs during the LLM request (e.g., if Ollama is unavailable), the function returns None instead of raising a critical error, allowing the main function to gracefully handle the situation and continue the chat.

3. The main Function and Interactive Loop:

The main function manages the user interface and the overall chat flow.

• Infinite Loop (while True): Creates a continuous chat session that persists until the user explicitly decides to exit.
• User Input: input("👤 You: ").strip() waits for user input. The .strip() method removes leading and trailing whitespace, preventing empty or malformed queries from being sent.
• Exit Commands: The check user_input.lower() in ("exit", "exit", "quit") allows the user to terminate the program using various exit command options.
• Empty Input: The check if not user_input: prevents empty messages from being sent to the model, prompting the user to enter a question.
• Interaction: main calls send_request_to_llm with the user's input and processes the received response, printing it to the console or reporting an error. The separation of responsibilities between main (interface management) and send_request_to_llm (model interaction) is a fundamental principle of good software design.

Controlling LLM Behavior via System Prompt

One of the most powerful tools when working with LLMs is the system prompt. It allows you to assign a specific role, communication style, or set of constraints to the model, significantly altering its responses without changing the model itself or its parameters. Experimenting with SYSTEM_PROMPT is an excellent way to understand how subtly you can tailor the assistant's behavior for specific tasks. Let's look at a few examples of how changing a single line can drastically impact the dialogue:

• `SYSTEM_PROMPT = "You are a sarcastic assistant. Respond with irony."` — The model will respond with humor and wit.
• `SYSTEM_PROMPT = "You are a strict teacher. Correct errors in questions."` — The assistant will not only answer but also point out grammatical or factual inaccuracies in queries.
• `SYSTEM_PROMPT = "Respond only in English. Always provide a code example."` — The model will strictly adhere to the English language and try to include code examples in its responses, if applicable.

These examples demonstrate how a system prompt transforms a static model into a flexible tool capable of adapting to various use cases. This is critically important for creating specialized chatbots, whether they are technical assistants, creative partners, or educational tools.

Key Takeaways

At this stage of development, our console LLM chat has achieved the following important results:

• Interactivity: The user can engage in a continuous dialogue, asking questions and receiving answers in real-time.
• Controlled Behavior: The system prompt allows for easy configuration of the model's role and communication style.
• Code Modularity: The LLM interaction logic is separated from the user interface logic, simplifying further development and debugging.
• Performance: Built-in response generation time measurement helps monitor the model's speed.
• Reliability: Basic error handling prevents program crashes due to connection or model issues.

Common Challenges and Solutions

When working with local LLMs and console applications, some common issues may arise:

• Program freezes after entering a question: This most often happens because Ollama is not running or the model has not yet fully loaded into memory. Ensure Ollama is actively running (ollama list in your terminal) and give the model time to initialize.
• Too brief responses from the model: If the SYSTEM_PROMPT specifies "answer concisely," the model will follow this instruction. For more detailed responses, modify the system prompt by removing this constraint or explicitly asking for "a detailed answer with examples."
• KeyboardInterrupt when pressing Ctrl+C: This is standard Python behavior. While not critical for a learning project, in production, you can wrap main() in a try/except KeyboardInterrupt block for a more graceful shutdown.
• Encoding issues (garbled characters in Windows): This can be resolved by setting the terminal encoding to UTF-8, for example, using the command $OutputEncoding = [System.Text.Encoding]::UTF8 in PowerShell.

Next Steps: Towards a Full-Fledged Dialogue

Although our console chat has become significantly more functional, it currently lacks "memory." Each request is processed independently, and the model doesn't remember previous dialogue turns. This means if you ask, "Can you do the same thing, but shorter?", the model won't understand what "the same thing" refers to. In the next part, we will explore how to implement message history, passing the entire dialogue context with each new request, which will transform it into a truly intelligent conversational partner.

— Editorial Team