# Running Local AI Models Without a GPU: Step-by-Step Client-Server Architecture Setup
Running large language models (LLM) locally without a graphics card is a realistic task for developers who prioritize privacy and data control. In this article, we'll walk through creating a client-server system where computations run on a second Linux computer, and the interface is accessible from your primary Windows machine. The emphasis is on security, minimizing resource usage, and leveraging existing hardware.
Preparing the "Compute Node": OS Selection and Disk Configuration
The key stage is setting up a dedicated computer without a GPU. Linux Mint (Debian-based) is ideal for this thanks to:
- Its minimalist Xfce interface, similar to Windows
- Low resource requirements
- Support for modern AI tools
Important installation notes:
- Use Ventoy to create a bootable USB drive (format into two partitions)
- During disk partitioning:
* Identify disks using the /sda, /sdb scheme
* Avoid Windows partitions (100 MB boot + 500 MB recovery)
* Select the ext4 file system
- Configure the bootloader for dual-boot: add a 5-10 second delay in BIOS
After installation, make sure to:
- Set up keyboard layout switching (left Shift + CapsLock in Xfce)
- Add a layout indicator to the panel
- Update the system via terminal:
sudo apt update && sudo apt upgrade -y
Deploying LMStudio: Working with GGUF Models
To run LLMs without a GPU, LMStudio (AppImage version) is the most flexible choice:
- Download the AppImage file from the official website
- Make it executable:
chmod +x LMStudio-*.AppImage - Launch via terminal:
./LMStudio-*.AppImage
Critical model settings:
- Set GPU-layers = 0
- Disable GPU caching
- Choose a model based on your RAM (7B-parameter versions recommended)
Verify it works in the Developer tab:
- Monitor prompt processing status
- Use the built-in chat for testing
- Enable the local server with the "Serve on Local Network" option
Setting Up Client-Server Interaction
To access the model from your main Windows computer, you'll need:
Server Side (Linux)
- Activate the API server in LMStudio on port 1234 (default)
- Configure the firewall:
sudo ufw allow 1234/tcp - Get your local IP:
hostname -I
Client Side (Windows)
- Install Open WebUI via Docker:
docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main- In the Open WebUI interface, specify:
- Base URL: http://[Linux_machine_IP]:1234/v1
- API Key: leave empty
Key Points: Core Deployment Principles
- Data Isolation: The model runs in a closed network without access to your main data
- Resource Optimization: Offloading to a second computer reduces strain on your workstation
- Configuration Flexibility: Easily switch models via LMStudio
- Connection Security: All requests stay within the local network; add TLS encryption if needed
Optimizing CPU Performance
To speed up LLM inference without a GPU:
- llama.cpp Configuration:
n_ctx = 4096
n_threads = [number_of_CPU_cores]
n_batch = 512- Optimized Models:
- Mistral-7B-Instruct-v0.2-GGUF (Q4_K_M)
- Phi-3-mini-4k-instruct-GGUF (Q5_K_S)
- Gemma-2B-it-GGUF (Q4_0)
- Load Monitoring:
- Use htop to track CPU usage
- Check memory usage with
free -h
Solving Common Issues
Issue: Slow response generation
Solution:
- Reduce n_ctx in the config
- Pick a Q4 quantized model over Q5
- Increase n_batch to 1024
Issue: Client fails to connect to server
Check:
- Firewall status on the Linux machine
- IP address accuracy in Open WebUI settings
- Server responsiveness via curl:
curl http://localhost:1234/v1/models
Issue: Keyboard layout switching not working
Fix:
- Open "System Settings" → "Keyboard"
- In the "Layouts" tab, set the Shift+CapsLock combo
- Add the indicator to the taskbar by right-clicking the panel
Conclusion: Practical Value of the Solution
This setup enables:
- Handling text tasks (translation, summarization) without cloud data transfers
- Repurposing old hardware effectively
- A flexible environment for testing various models
Key takeaway: You can build a functional LLM infrastructure without a GPU by focusing on proper environment setup and optimized models. The system scales effortlessly — just tweak GPU-layers when adding a graphics card, no architecture overhaul needed.
— Editorial Team
No comments yet.