Retrieval-Augmented Generation with LangChain

In this demo, you’ll use DeepEval, a popular open-source LLM evaluation framework. It has a simple and intuitive set of APIs you’ll soon use to assess SportsBuddy. Open your Jupyter Lab instance with the following command:

jupyter lab

Install DeepEval with:

pip install -U deepeval

You’ll first test the retrieval component. Create a new Python file called deepeval-sportsbuddy-test.py. Import DeepEval classes for contextual precision, recall, and relevance:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
  ContextualPrecisionMetric,
  ContextualRecallMetric,
  ContextualRelevancyMetric
)

contextual_precision = ContextualPrecisionMetric()
contextual_recall = ContextualRecallMetric()
contextual_relevancy = ContextualRelevancyMetric()

Next is to create a test case. A DeepEval test case is as simple as creating an instance of LLMTestCase and running your desired metrics on it. Because you’ll be evaluating SportsBuddy, open this lesson’s starter project in Jupyter Lab. Here, you’ll see the question and response. Visit the 2024 Summer Olympics Wikipedia page to get the retrieved context relevant to the question. Back in your Python file, create the test case:

test_case = LLMTestCase(
  input="Which programmes were dropped from the 2024 Olympics?",
  actual_output="Four events were dropped from weightlifting for the 
    2024 Olympics. Additionally, in canoeing, two sprint events 
    were replaced with two slalom events. The overall event 
    total for canoeing remained at 16.",
  expected_output="Four events were dropped from weightlifting.",
  retrieval_context=[
    """Four events were dropped from weightlifting."""
 ]
)

An LLMTestCase requires your query, the RAG’s output, your expected output so DeepEval has a good reference point, and a retrieval context so DeepEval has a good idea of the kind of context your RAG used to provide its answer. Pretty straightforward. Pass the test case to all three metrics for evaluation:

evaluate(
  test_cases=[test_case],
  metrics=[contextual_precision, contextual_recall, contextual_relevancy]
)

Return to your terminal and run the file with the following command:

python deepeval-sportsbuddy-test.py

Here’s this test’s result:

======================================================================

Metrics Summary

  - ✅ Contextual Precision (score: 1.0, threshold: 0.5, strict: False, 
    evaluation model: gpt-4o, reason: The score is 1.00 because the 
    context directly answers the question by stating 'Four events 
    were dropped from weightlifting.' Great job!, error: None)
  - ✅ Contextual Recall (score: 1.0, threshold: 0.5, strict: False, 
    evaluation model: gpt-4o, reason: The score is 1.00 because the 
    expected output perfectly matches the content in the first node 
    of the retrieval context. Great job!, error: None)
  - ❌ Contextual Relevancy (score: 0.0, threshold: 0.5, strict: False, 
    evaluation model: gpt-4o, reason: The score is 0.00 because the
    context only mentions 'Four events were dropped from weightlifting' 
    without specifying which programmes or providing a comprehensive 
    list of dropped programmes from the 2024 Olympics., error: None)

For test case:

  - input: Which programmes were dropped from the 2024 Olympics?
  - actual output: Four events were dropped from weightlifting for 
    the 2024 Olympics. Additionally, in canoeing, two sprint events 
    were replaced with two slalom events. The overall event total 
    for canoeing remained at 16.
  - expected output: Four events were dropped from weightlifting.
  - context: None
  - retrieval context: ['Four events were dropped from weightlifting.']

======================================================================

Overall Metric Pass Rates

Contextual Precision: 100.00% pass rate
Contextual Recall: 100.00% pass rate
Contextual Relevancy: 0.00% pass rate

======================================================================

It looks like a lot, but it’s simple. The Metrics Summary section shows the type of metric, the parameters you used, the score, and the reason for the score. Here’s what each item means:

• score: The overall score. It ranges from 0 to 1 and is affected by the threshold and strict parameters.
• threshold: A float value that defaults to 0.5. Any score below it is a fail, and any value above it is a pass.
• strict: A Boolean value that forces a binary score. That’s a 1 for pass or 0 for fail. When set to false, the score can range between 0 and 1. It’s false by default. When true, it overrides the threshold, setting it to 1.
• evaluation model: Defaults to gpt-4o. This refers to the LLM DeepEval uses to evaluate the metric. You can specify your custom LLM if you wish.
• reason: A reason for the given score.

From the results above, precision and recall were great. But contextual relevance wasn’t. This could mean your given context didn’t have enough depth for your RAG to give you a detailed response or your question lacked some clarity. In this case, it might be both. The given context indeed had very little information about the question. And the question mentions “programmes” when the right terminology should be “events.” This immediately gives a clue as to which part of your RAG could need some attention.

Now, on to some generation metrics. For the generation component, you’ll measure the answer relevancy and faithfulness metrics. Return to your Python file and add the following:

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

answer_relevancy = AnswerRelevancyMetric()
faithfulness = FaithfulnessMetric()

evaluate(
  test_cases=[test_case],
  metrics=[answer_relevancy, faithfulness]
)

Rerun the script and check the result:

=====================================================================

Metrics Summary

  - ✅ Answer Relevancy (score: 0.6666666666666666, threshold: 0.5, 
    strict: False, evaluation model: gpt-4o, reason: The score is 0.67 
    because while the response contains relevant information, it veers 
    off-topic by discussing the overall event total for canoeing, 
    which does not directly answer the specific question about which 
    programmes were dropped from the 2024 Olympics., error: None)
  - ✅ Faithfulness (score: 1.0, threshold: 0.5, strict: False, evaluation 
    model: gpt-4o, reason: The score is 1.00 because there are no 
    contradictions, indicating a perfect alignment between the actual 
    output and the retrieval context. Great job maintaining accuracy!,
    error: None)

For test case:

  - input: Which programmes were dropped from the 2024 Olympics?
  - actual output: Four events were dropped from weightlifting for 
    the 2024 Olympics. Additionally, in canoeing, two sprint events 
    were replaced with two slalom events. The overall event total 
    for canoeing remained at 16.
  - expected output: Four events were dropped from weightlifting.
  - context: None
  - retrieval context: ['Four events were dropped from weightlifting.']

======================================================================

Overall Metric Pass Rates

Answer Relevancy: 100.00% pass rate
Faithfulness: 100.00% pass rate

======================================================================

For answer relevance, you got just about two-thirds of the full score. DeepEval does a great job by giving you the reason for this score. It says that although the answer is okay, it introduces other information that slightly digresses from the question. It would have scored higher if it stayed on the topic or, better still, left out the extra information.

SportsBuddy, however, appears to be faithful, at least for this test. All these constitute one test. You’ll have to run a good number of tests to get a good overview of the state of your RAG. These tests were intuitive enough and simple.

Up next, you’ll learn about query analysis.