Message history in Python Ollama LLM chat

Implementing Message History and Context in a Local LLM Chat with Python

The console chat based on LiteLLM and Ollama from the previous part processes requests independently. The model cannot see previous messages, which breaks the dialogue: references to past responses are ignored. The solution is to store message history in a Python list and pass it with each new request.

This turns a sequence of calls into a coherent conversation. The model receives full context: the system instruction, past user/assistant pairs, and the current input.

Message Structure for Context

The message format is standard for OpenAI-compatible APIs:

• system: a persistent instruction, added separately
• user: the user's query
• assistant: the model's response

History contains only user/assistant pairs. When sending:

[
    {"role": "system", "content": SYSTEM_PROMPT},
    *conversation_history,
    {"role": "user", "content": user_message}
]

Order is critical: system sets behavior, history provides context, new user is the current query.

Updated main.py Code

Full script with history and context trimming:

# -*- coding: utf-8 -*-
import time
from typing import Optional
from litellm import completion

MODEL = "ollama_chat/qwen2.5:3b"
API_BASE = "http://localhost:11434"
SYSTEM_PROMPT = "You are a helpful assistant. Respond concisely and to the point."

MAX_HISTORY_MESSAGES = 6


def trim_history(history: list, limit: int) -> list:
    if len(history) <= limit:
        return history
    return history[-limit:]


def send_request_to_llm(user_message: str, history: list) -> Optional[str]:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        *history,
        {"role": "user", "content": user_message},
    ]
    try:
        start_time = time.time()
        response = completion(
            model=MODEL,
            messages=messages,
            api_base=API_BASE,
            request_timeout=120,
        )
        duration = time.time() - start_time
        print(f"\nGeneration time: {duration:.2f} sec")
        return response.choices[0].message.content
    except Exception as e:
        print(f"\nRequest error: {e}")
        return None


def main() -> None:
    print("Local AI assistant with memory started.")
    print("Enter a question or 'exit' to quit.\n")

    conversation_history = []

    while True:
        user_input = input("You: ").strip()

        if user_input.lower() in ("exit", "quit"):
            print("Goodbye!")
            break

        if not user_input:
            print("Please enter a question.")
            continue

        print("\nModel thinking...")
        answer = send_request_to_llm(user_input, conversation_history)

        if answer is not None:
            print(f"\nAI: {answer}\n")
            conversation_history.append({"role": "user",      "content": user_input})
            conversation_history.append({"role": "assistant", "content": answer})
            conversation_history = trim_history(conversation_history, MAX_HISTORY_MESSAGES)
        else:
            print("\nFailed to get a response. Check if Ollama is running.\n")


if __name__ == "__main__":
    main()

Key Logic Changes

• conversation_history: list in main() stores message pairs
• send_request_to_llm: accepts history, expands via *
• Saving pairs: after a response, user and assistant are added
• trim_history: trims to MAX_HISTORY_MESSAGES (6 items — 3 dialogues)

| Component | Before Update | After Update |

|-----------|---------------|--------------|

| messages | system + user | system + history + user |

| Memory | None | List in RAM |

| Size | Unlimited | Up to 6 messages |

Trimming prevents context overflow: old messages increase tokens and slow down inference.

Testing Context

Dialogue with dependent queries:

• You: Name three Python web development frameworks
• AI: Django, Flask, FastAPI
• You: Tell me more about the second one
• AI: Flask is a lightweight microframework...

The model correctly references "the second" by seeing the previous response.

Limitations of the Current Implementation

• History in RAM: lost on restart
• Fixed limit: 6 messages (configurable)
• No persistence: requires Redis/DB for production

For a Telegram bot or web app, the logic remains: history in user session.

What's Important

• Message format: system separately, history only user/assistant
• History trimming: [-limit:] preserves recent pairs
• Separation of responsibilities: LLM requests in send_request_to_llm, state in main()
• Timeouts: request_timeout=120 for long contexts
• Logging: generation time for performance debugging

— Editorial Team