Assessing a RAG Pipeline

Assessing a RAG Pipeline

A RAG app has two main components: the retrieval component and the generation component. The former retrieves dynamic data from some data source such as a website, text, or database. The generation component combines the retrieved data with the query to generate a response with an LLM. Each of these components consists of smaller moving parts. Considering all these components and their subcomponents, it’s accurate to call the RAG process a chain or a pipeline.

The reality is that a pipeline’s performance is largely determined by the performance of its weakest component. If your Internet Service Provider (ISP) gives you internet at 10Gbps but you have a 5Gbps router, you’re not going to get beyond 5Gbps even if your ethernet port is 10Gbps. To make the most of what your ISP offers, you have to assess each of your network components and enhance the ones that fall short.

It might be that you need to refine your queries, prompts, embedding model, vector store, retrieval search algorithm, response generation, ranking, or something else. Read on to identify key subcomponents and metrics that can help you enhance your RAG app.

Assessing the Retriever Component

Many parameters control a retriever’s output. The retrieval phase begins with loading the source data. How quickly is data loaded? Is all desired data loaded? How much irrelevant data is included in the source? For media sources, for instance, what qualifies as unnecessary data? Will you get the same or better results if, for example, your videos were compressed?

Next is embedding the data. A good embedding results in an excellent representation of the data in vector spaces. It also uses less space and processes data quickly. Other things to consider are how well the embedding model captures semantics, keywords, and contexts or domains. For instance, an embedding model used in the healthcare sector should be able to understand a virus differently from one used in technology. This could lead to erroneous results.

The embedding model also takes information in chunks. You can’t possibly dump gigabytes of data at once. Therefore, parameters like the chunk size and how much text from one chunk flows into the other can all impact the model’s performance. You’ll even have to ensure that your embedding model receives all the data you feed it.

The next immediate consideration is how well the model does with search. If the embedding model didn’t align the data correctly, its search will likely perform poorly, too. There are different types of search, as you saw in the previous lesson. A hybrid search, for instance, generally gives better responses. But at what cost?

Closely related to search performance is re-ranking. Re-ranking aims to enhance search results. However, when done poorly, re-ranking — with filtering or compression — can also exclude relevant data and use more system resources.

Assessing the Generator Component

The story is similar for the generator component. Many parameters significantly affect its performance. There’s the temperature, which controls the randomness or creativity of the LLM. It ranges between 0 and 1. 0 means it sticks to the given context strictly, and 1 means it has the freedom to respond with whatever it thinks is suitable to your question.

Prompt templates exist because some specific sentence constructions yield more accurate results than others. Many techniques seek to tackle prompt effectiveness. Some techniques regenerate “better” prompts from your initial prompt before accessing the LLM. Prompt engineering is the process of designing and refining input prompts to optimize the performance and output of LLMs. It focuses on clarity, context, precision, constraints, and iterations. There are dedicated teams and efforts targeted at perfecting prompts for the best LLM outputs.

LLMs work with tokens, the fundamental unit of data they operate on. Paid LLMs charge based on tokens. Some LLMs can process more tokens at a time, while others are limited. As of this writing, Bing Chat can handle up to 32,000 tokens in a single interaction. For paid services, this could mean you’re paying more depending on how frequently you chat with the LLM, how much data forms your prompt, and how much data the LLM’s response contains. In this case, you might want to look for offline, free LLMs or offline versions, which also have their quirks.

LLMs are not all created equal. Some have better training data, more recent training data, varying levels of bias, and differing amounts of contextual data. For instance, GPT-3 has about 175 billion parameters whereas LLaMA 3 has models with 8 billion, 70 billion, and 405 billion parameters. More parameters generally mean larger datasets, which could affect query speed but also produce more relevant responses.

Evaluation Metrics

Due to the complex, integrated nature of RAG systems, evaluating them is a bit tricky. Because you’re dealing with unstructured textual data, how do you assess a scoring scheme that reliably grades correct responses? Consider the following prompts and their responses:

Prompt:
"What is the capital of South Africa?"

Answer 1:
"South Africa has three capitals: Pretoria (executive), Bloemfontein (judicial), 
  and Cape Town (legislative)."

Answer 2:
"While Cape Town serves as the legislative capital of South Africa, Pretoria 
  is the seat of the executive branch, and Bloemfontein is the judicial capital."

Both answers are essentially the same in meaning yet very different in how the sentences are constructed. A good metric and evaluation framework should be able to score full marks for both answers above. This is very different from quantitative analysis, which almost always gives you specific figures or a valid range by which you could easily tell if an answer was right or wrong.

Consider the following, too:

Prompt:
"What was the cause of the American Civil War?"

Answer 1:
"The primary cause of the American Civil War was the issue of slavery, 
  specifically its expansion into new territories."

Answer 2:
"While states' rights and economic differences played roles, the main 
  cause of the American Civil War was the debate over slavery and its expansion."

Both answers above are relatively similar in terms of sentence construction and even the words used. However, the second answer is misleading and should score low marks during evaluation. There are also instances where your RAG could generate responses that are factual but not relevant to the given context. Or the answer might be correct but vague, which means it’s of little use.

Exploring RAG Metrics

Over the years, several useful metrics have emerged, targeting different aspects of the RAG pipeline. For the retrieval component, common evaluation metrics are nDCG (Normalized Discounted Cumulative Gain), Recall, and Precision. nDCG measures the ranking quality, evaluating how well the retrieved results are ordered in terms of relevance. Higher scores are given for relevant results that appear at the top. Recall measures the model’s ability to retrieve relevant information from the given dataset. Precision measures how many of the search results are relevant. For best results, use all metrics. Other kinds of metrics available are LLM Wins, Balance Between Precision and Recall, Mean Reciprocal Rank, and Mean Average Precision.

For the generation component, common metrics include Faithfulness and Answer Relevance. Faithfulness measures the correctness of the response based on the retrieved context. It’s concerned with answers that stem from the retrieved information and nothing else. A fact, in this sense, is that which is available in the retrieved context. It doesn’t matter that the retrieved context might hold inaccurate information. Consider a situation in which the source data contains a text that says, “Cristiano Ronaldo is the best footballer ever and has the most Ballon d’Or.” Irrespective of the fact that this isn’t true, a faithfulness measure should score full marks for your RAG if it returns this answer in response to a query like, “Which footballer has the most Ballon d’Or?”

Answer relevance assesses how well the generated response addresses the user’s question. This metric scores high marks for complete answers and answers that don’t contain repetition or redundancy. Its working principle is primarily reverse engineering. That is, the LLM should be able to regenerate the question from the given answer.

Other metrics available for the generation component are Bilingual Evaluation Understudy, Metric for Evaluation of Translation with Explicit Ordering, and Recall-Oriented Understudy for Gisting Evaluation. Much research is ongoing in the entire AI ecosystem, which could mean newer and better RAG performance and metrics in the future. In the meantime, you need to use existing tools to help improve your RAG app. In the next section, you’ll assess some evaluation tools.

Evaluating RAG Evaluation Tools

Just as there’s no shortage of RAG evaluation metrics, there’s equally a good number of evaluation tools. Some use custom metrics not previously mentioned, and proprietary metrics, too. Depending on your use case, one or a combination of specific metrics will boost your RAG’s performance significantly. Examples of RAG evaluation frameworks include Arize, Automated Retrieval Evaluation System (ARES), Benchmarking Information Retrieval (BEIR), DeepEval, Ragas, OpenAI Evals, Traceloop, TruLens, and Galileo.

DeepEval is an open-source LLM evaluation framework. This means it’s free to use. With DeepEval, you evaluate RAGs by executing test cases. You provide the prompt, the generated response, and the expected answer. You follow this procedure to evaluate both retrieval and generation components of your RAG app.

For retrieval component evaluation, DeepEval offers tools for assessment using contextual precision, recall, and relevance. As earlier indicated, you need to measure all three of these metrics to gain a better appreciation for how your RAG app performs.

In the next demo, you’ll evaluate SportsBuddy with two different evaluation tools: DeepEval and Ragas.