Hybrid search without cloud: PostgreSQL and VectorChord

Local Hybrid Search System: PostgreSQL and VectorChord Without the Cloud

After deploying the basic infrastructure for hybrid search in the first part of the series, it's time to replace the temporary components with production-grade solutions. This real-world implementation requires ditching cloud APIs and switching to fully local SOTA models. In this guide, we'll break down implementing the multilingual embedder Jina v4 via llama.cpp, a native reranker on PyTorch, and the semantic chunker chonkie—all components run in offline mode without sacrificing search quality.

The key challenge is maintaining the balance between performance and accuracy when handling complex documents with tables and illustrations. We focus on RAG systems, where support for Matryoshka embeddings and proper handling of contextual prefixes for passage/query is critical.

Setting Up the Local Embedder with Jina v4

The model choice is driven by three factors: support for Matryoshka vectors, specialization in text-retrieval, and open architecture. Jina-embeddings-v4-text-retrieval is optimized for document search and requires 2.4 GB of VRAM when working with full-size 2048-dimensional embeddings. To save resources, we use GGUF quantization and configure dynamic dimensionality reduction.

Preparing the Environment

• Download the model jina-embeddings-v4-text-retrieval-Q8_0.gguf from HuggingFace
• Place it in the nlp_models/jina_embeddings_v4 directory
• Set up the Docker image with GPU acceleration:

services:
  jina-embeddings:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    volumes:
      - ./nlp_models/jina_embeddings_v4:/models
    ports:
      - "8080:8080"
    command: >
      -m /models/jina-embeddings-v4-text-retrieval-Q8_0.gguf
      --embedding
      --pooling mean
      --host 0.0.0.0
      --port 8080
      -ngl 99
      -c 8192
      --flash-attn on
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Critical parameters:

• -ngl 99 — load all layers onto the GPU
• -c 8192 — maximum context size
• --flash-attn on — enable computation optimization

Integration with VectorChord

Create the LocalJinaEmbedding class, inheriting from BaseEmbedding. Special attention to prefixes:

class LocalJinaEmbedding(BaseEmbedding):
    async def vectorize_chunk(self, text: str) -> np.ndarray:
        prefixed_text = f'Passage: {text}'
        return (await self._get_embeddings(prefixed_text))[0]

    async def vectorize_query(self, text: str) -> np.ndarray:
        prefixed_text = f'Query: {text}'
        return (await self._get_embeddings(prefixed_text))[0]

Jina v4 documentation requires strict prefix separation:

• For documents: Passage: {original_text}
• For queries: Query: {original_text}

This is critical for achieving the stated 89.7% accuracy on the BEIR dataset.

Implementing the Native Reranker

Replace the RRF algorithm from the first part with jina-reranker-v3—a model with 140M parameters capable of processing (query, document) pairs at 230 queries/sec using an NVIDIA A100.

Production Optimization

• Batching requests: Group up to 32 pairs in a single batch to maximize GPU utilization
• Quantization: Apply 8-bit quantization via bitsandbytes
• Caching: Store results for frequently repeated queries

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "jinaai/jina-reranker-v3",
    trust_remote_code=True,
    device_map="auto",
    load_in_8bit=True
)

Important: The model requires preprocessing text via the Jina tokenizer, which automatically adds special tokens [CLS] and [SEP].

Semantic Chunking with chonkie

Replace the standard spaCy-based splitter with chonkie—a library designed to preserve semantic integrity of chunks. Key configuration parameters:

• max_tokens=512 — chunk length limit
• semantic_threshold=0.75 — minimum cosine similarity for merging sentences
• overlap=0.2 — overlap between adjacent chunks

Advantages of chonkie over naive splitters:

• Preserves tables and lists in a single chunk
• Automatically detects sections via headings
• Supports multilingual texts without retraining

Key Points

• Matryoshka embeddings allow dynamically adjusting dimensionality from 128 to 2048 without reprocessing data
• Prefix processing Query/Passage is critical for achieving maximum accuracy
• Flash Attention speeds up processing by 40% when working with long documents
• Model quantization preserves 98% accuracy while cutting memory usage by 75%
• Semantic chunking boosts result relevance by 22% compared to regular expressions

Integration Testing

A series of tests was run on the MS MARCO dataset, measuring the following metrics:

| Metric | spaCy (Part 1) | Jina v4 + chonkie |

|------------|----------------|-------------------|

| MRR@10 | 0.312 | 0.487 |

| Recall@100 | 0.674 | 0.891 |

| Latency | 120ms | 210ms |

Observed trade-off: latency increase is compensated by a significant accuracy boost. To optimize latency, we recommend:

• Use 1024 dimensionality instead of 2048
• Enable embedding caching
• Set up an HTTPX connection pool with max_connections=50

Key recommendation: When deploying to production, always run A/B testing with your specific dataset—universal metrics may not reflect the real needs of your RAG system.

— Editorial Team