RAG Systems: Enhancing LLM Answer Accuracy with Rerankers
Large Language Models (LLMs) demonstrate impressive text generation capabilities, yet their propensity for "hallucinations"—producing false or unverified information—remains a significant challenge. To address this, the Retrieval-Augmented Generation (RAG) architecture is widely employed. RAG enables LLMs to draw upon external, verified data sources, substantially improving the trustworthiness and relevance of generated responses. This article delves into the operational principles of RAG, examining key components such as embeddings, vector databases, retrievers, and particularly focusing on the crucial role of rerankers in balancing search speed with information accuracy.
RAG Fundamentals: From Hallucinations to Facts
The problem of LLM hallucinations stems from models being trained on vast datasets, enabling them to generate plausible-sounding text even when lacking specific knowledge on a requested topic. RAG systems tackle this by providing the model access to up-to-date and relevant information from an external knowledge base. The process involves two main stages: Retrieval and Generation. During the retrieval phase, the system extracts text snippets from the database that are most relevant to the user's query, then passes them to the LLM as additional context for generating a response.
For effective use of external data, it must be prepared. Long documents are broken down into smaller, meaningful segments, or chunks. This is essential because LLMs have a limited context window, meaning the maximum amount of text they can process at once. To maintain information coherence between adjacent chunks, an overlap technique is used, where a small portion of the preceding segment is included at the beginning of the next. This helps the model better understand the context when working with individual text parts.
Vector Databases and Embeddings: Semantic Search
At the heart of a RAG system lies the mechanism for finding relevant data fragments. Computers cannot directly understand the meaning of words; they operate with numbers. Therefore, texts are transformed into numerical representations called embeddings or vectors. An embedding is a multi-dimensional vector (a sequence of numbers) that encodes the semantic meaning of a word, phrase, or an entire text segment. Words or phrases similar in meaning will have vectors located close to each other in a multi-dimensional space.
The process of creating embeddings is carried out using specialized embedding models, which are trained on vast text corpora. These models learn to map words and their context to unique numerical representations. For instance, if you take the vectors for "king," "man," and "woman," the vector for "queen" will be very close to the result of the operation "vector(king) - vector(man) + vector(woman)." This demonstrates embeddings' ability to capture complex semantic and syntactic relationships in language.
The generated embeddings are stored in vector databases. These databases are optimized for rapid nearest-neighbor searches. When a user poses a question, their query is also transformed into an embedding, and then the vector database finds text fragments whose embeddings are most similar to the query's embedding. Vector similarity is typically measured using metrics such as cosine similarity.
Here is an example Python code demonstrating embedding creation using the OpenAI API:
# To run this example, you need to install the openai library and obtain an API key
from openai import OpenAI
import os
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
text = "A fluffy cat named Whiskers enjoys telling tales of ancient explorers."
response = client.embeddings.create(
model="text-embedding-3-small",
input=text,
dimensions=768 # explicitly set the dimensionality
)
vector = response.data[0].embedding
print(f"Vector dimensionality: {len(vector)}") # will output 768
print("First 20 numbers:", vector[:20])
print("... and so on up to the 768th number")This code transforms a textual phrase into a numerical vector of 768 elements, which can then be used for searching in a vector database.
Retrievers and Scaling: Performance Challenges
A retriever is a component of a RAG system responsible for extracting an initial set of relevant documents or fragments from a vector database. It uses the user query's embedding to find the most similar embeddings of stored chunks. For small document collections, a retriever can perform a brute-force search, comparing the query embedding with all stored vectors. However, as data volumes grow (billions of fragments), this approach becomes inefficient and too slow.
To address the scaling problem, various Approximate Nearest Neighbor (ANN) algorithms are employed. These algorithms create special indexes, such as IVF (Inverted File Index), HNSW (Hierarchical Navigable Small World), or LSH (Locality Sensitive Hashing). ANN indexes significantly speed up searches by sacrificing a small degree of precision. They find not perfectly exact, but very close neighbors in much less time. Thus, a trade-off emerges between recall (completeness, i.e., the ability to find all relevant items) and latency (delay, i.e., response speed).
Despite speed optimizations, the results returned by a retriever are not always ideal. Retrievers might extract fragments that are semantically close to the query but are not the most relevant or precise for forming the final answer. This is because the embeddings used for search are often unidirectional (bi-encoder), meaning the query and documents are encoded independently, and then their vectors are compared. While efficient for quick searches, this approach can miss subtle contextual connections.
Boosting Accuracy with Rerankers
To overcome the limitations of retrievers and enhance the accuracy of RAG system responses, rerankers are employed. A reranker is an additional component that receives a small set (e.g., 10-100) of potentially relevant fragments from the retriever and reorders them based on a deeper relevance assessment. It acts as a filter, discarding less useful results and elevating the most significant ones to the top.
Unlike the embedding models used by retrievers, rerankers often utilize cross-encoders. Cross-encoders take "query-document" pairs as input and process them jointly, evaluating their mutual relevance. This allows them to capture more complex and nuanced interactions between the query and context than simple vector comparisons. Although cross-encoders are significantly slower than bi-encoder embedding models, they are applied to an already filtered, small set of documents, making their use computationally feasible.
Thus, rerankers help strike an optimal balance between speed and accuracy. A fast retriever with ANN indexes quickly selects a broad range of potentially relevant chunks, and then a slower but more precise reranker re-evaluates this limited set, ensuring the LLM receives the highest quality context for generating a response. This multi-stage approach significantly improves the quality and trustworthiness of information provided by a RAG system, minimizing the risk of hallucinations and enhancing overall user satisfaction.
Key Takeaways
- RAG systems enable LLMs to leverage external data for generating accurate and reliable answers, combating the problem of "hallucinations."
- Embeddings transform text into numerical vectors that encode semantic meaning, facilitating similarity searches in vector databases.
- Retrievers utilize embeddings and ANN indexes to quickly extract an initial set of relevant fragments, striking a balance between speed and recall.
- Rerankers (often based on cross-encoders) re-evaluate retriever results, filtering out less relevant fragments and improving the accuracy of context for the LLM.
- The combination of a fast retriever and an accurate reranker achieves an optimal balance between performance and quality in RAG systems.
— Editorial Team
No comments yet.