Local RAG in Obsidian with Ollama and Gemma

Local RAG Assistant in Obsidian: Ollama, Gemma 4, and Infio Copilot

Using Obsidian, Ollama, and the Infio Copilot plugin, you can build a RAG system that indexes markdown notes and answers questions about them entirely offline—no cloud required. It uses the bge-micro-v2 model for embeddings and local LLMs like gemma4:e2b for generation. Indexing a 70 MB knowledge base takes just 1–2 minutes, with responses generated in 15+ seconds on an RTX 3060 Ti with 8 GB VRAM.

The system separates the embedding model for text vectorization from the LLM for response generation. This optimizes speed: compact bge-micro-v2 for embeddings, and a more powerful gemma4:e2b for answers.

Why Obsidian's Standard Search Falls Short

Basic keyword search misses semantically related snippets. This RAG setup fixes that with vector search: text is split into chunks, converted to embeddings, and stored in a local index. Your query gets vectorized too, pulling the most relevant chunks as context for the LLM.

Issues with previous approaches:

• Qwen models in Ollama are sluggish on 8 GB VRAM due to RAM swapping.
• Smart Connections: fast search, but chat is paywalled.
• Original Copilot: painfully slow indexing via Ollama embeddings (up to an hour).

Installing Ollama and Models

Ollama runs LLMs locally. Download and test:

ollama pull gemma4:e2b
ollama run gemma4:e2b

gemma4:e2b is twice as fast as qwen3:8b in real-world use. qwen3.5:9b and qwen3.5:4b are alternatives for lower VRAM, but with some quality trade-offs.

On a GPU with 8 GB, the model fits entirely. Without a GPU or on weaker hardware, speeds drop dramatically.

BRAT and Installing Infio Copilot

Infio Copilot is a Copilot fork with built-in bge-micro-v2 embeddings. Install via BRAT:

• In Obsidian: Settings → Community Plugins → Browse → install BRAT.
• In BRAT: Add Beta Plugin → Infio Copilot repository.
• In Infio settings: Ollama provider, http://localhost:11434, model gemma4:e2b.

Indexing: automatic chunking with bge-micro-v2, vector index built in minutes.

Performance Comparison

| Component | Smart Connections | Copilot | Infio Copilot |

|-----------------|-----------------------|---------------|-------------------|

| Embeddings | bge-micro-v2 (built-in) | Ollama | bge-micro-v2 (built-in) |

| Indexing 70 MB | 1–2 min | up to 1 hr | 1–2 min |

| Chat | paywalled | free, slow | free, fast |

| LLM | cloud | Ollama | Ollama |

Infio is the sweet spot: fast vectorization + local generation.

Hands-On: Queries and Responses

Example: A question on your notes pulls relevant chunks + a synthesized answer from gemma4:e2b. Stability is decent—sometimes start a new chat. No fine-tuned chunking options, but defaults work well.

Developer perks:

• Semantic search across chats/notes (architecture, code, research).
• Total data privacy.
• Free once set up.

Limitations:

• VRAM dependency (8 GB minimum for smooth sailing).
• LLM inconsistency.
• Initial setup time.

Key Takeaways

• Embeddings and LLM are separate: bge-micro-v2 for vectors, gemma4:e2b for generation.
• Fast indexing only with built-in embeddings.
• On 8 GB VRAM, gemma4:e2b delivers usable speed (15+ sec/response).
• Ideal for personal knowledge bases, not production.
• BRAT required for Infio Copilot.

— Editorial Team