LLM Dialog: Creating a Go Utility for Comparing Language Models

Automated LLM Dialogs: Building a Go GUI Utility to Compare Language Models

Automating conversations between two language models unlocks exciting possibilities for testing, comparing, and analyzing AI behavior. This Go-based GUI utility, powered by Fyne, visualizes the process, giving researchers and developers a handy tool for working with local LLMs via Ollama.

App Architecture for LLM Dialogs

The program is a Fyne-based desktop app with an intuitive interface for setting up and controlling conversations. The left panel features controls: pick two models from the ollama list output, optionally assign roles, enter a conversation topic, set round limits, and timeouts in minutes. The right side shows the chat flow with color-coded messages and timestamps.

Key technical highlights:

• Leverages local Ollama server on port 11434
• HTTP API for model interactions
• Per-model context management with the last 20 messages
• Timeout handling via Go contexts to prevent hangs
• Reset mechanism to clear conversation history

Implementing Model Interactions

The app takes a structured approach to conversation management. Each model maintains its own context, including a system message with its role and chat history, ensuring coherent exchanges without context bloat.

Conversation flow:

• Initialize contexts for both models with system prompts
• Alternate turns for the set number of rounds
• Update histories for both models after each response
• Enforce timeouts via context

Here's sample code for calling the Ollama API:

func callOllamaAPI(ctx context.Context, model string, messages []Message) (string, error) {
    requestBody := ChatRequest{
        Model:    model,
        Messages: messages,
        Stream:   false,
    }
    requestBody.Options.NumPredict = 1024
    
    jsonData, err := json.Marshal(requestBody)
    if err != nil {
        return "", fmt.Errorf("request marshal error: %v", err)
    }
    
    req, err := http.NewRequestWithContext(ctx, "POST",
        "http://localhost:11434/api/chat",
        bytes.NewBuffer(jsonData))
    if err != nil {
        return "", fmt.Errorf("request creation error: %v", err)
    }
    req.Header.Set("Content-Type", "application/json")
    
    client := &http.Client{Timeout: 0}
    resp, err := client.Do(req)
    if err != nil {
        return "", fmt.Errorf("HTTP request error: %v", err)
    }
    defer resp.Body.Close()
    
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return "", fmt.Errorf("response read error: %v", err)
    }
    
    var response ChatResponse
    if err := json.Unmarshal(body, &response); err != nil {
        return "", fmt.Errorf("JSON parse error: %v", err)
    }
    return response.Message.Content, nil
}

Practical Use Cases

This tool shines for tech pros in several ways:

• Model Comparison

- Test LLMs on identical prompts

- Spot differences in style, tone, and content

- Analyze consistency and contradictions in chats

• Role Impact Studies

- Assign personas to models

- Observe behavior shifts by role

- Experiment with AI social dynamics

• Edge Case Testing

- Check stability in long conversations

- Probe handling of tricky or conflicting topics

- Evaluate context retention

• Education and Demos

- Showcase LLM mechanics

- Visualize text generation

- Explore AI app design patterns

Implementation Details

Built with:

• Fyne for the GUI
• Go standard library for networking and JSON
• Ollama API for LLMs
• Goroutines and mutexes for concurrency
• Contexts for timeout control

Core data structures:

• Message for individual messages
• ChatRequest and ChatResponse for API calls
• ModelContext for per-model state
• ConversationState for chat tracking
• guiApp as the main app struct

Optimization and Enhancements

To boost performance and features:

• Cache model responses for repeats
• Add real-time streaming
• Export chats to JSON, Markdown, PDF
• Integrate dialog quality metrics
• Support generation params (temperature, top_p)
• Load custom prompts from files

Key Takeaways

• Automates head-to-head LLM chats without human input
• Fyne GUI makes tweaking settings a breeze
• Local Ollama support means no cloud dependency
• Smart context and history management keeps things coherent
• Ideal for AI comparisons, testing, and research

— Editorial Team