AI · Nov 17, 2026 · 12 min read

LLM Evaluation Pipeline with RAGAS, DeepEval & LangSmith

Build a production-grade LLM evaluation system using RAGAS for retrieval metrics, DeepEval for G-Eval and custom judges, and LangSmith for tracing. Includes complete Python code, a worked RAG example,

LLM Evaluation Pipeline with RAGAS, DeepEval & LangSmith

LLM Evaluation Challenges in Real Applications

LLM-powered features deployed to production frequently generate outputs that read fluently yet contain subtle hallucinations, ignore key details from retrieved context, or address a tangential interpretation of the user query. These issues remain invisible to standard test suites because LLM outputs occupy an open-ended space where boolean pass-fail checks cannot be written in advance.

Systematic evaluation supplies the missing feedback mechanism. It quantifies retrieval quality, generation faithfulness, and answer relevance on both static test sets and sampled live traffic. Without it, product teams iterate on prompts and retrieval strategies based on anecdotal user feedback that arrives too late and at too small a sample to drive reliable decisions.

RAGAS supplies metrics that isolate retrieval from generation and uses LLM-as-a-judge to score each component independently. DeepEval complements this with G-Eval, which feeds an LLM a detailed rubric, chain-of-thought instructions, and example judgments before producing a score accompanied by an explanation trace. LangSmith records every trace, prompt version, and resulting metric so that changes can be compared quantitatively rather than subjectively. Together these tools convert evaluation from an occasional manual review into an automated, version-controlled part of the development workflow.

The pipeline described below was built for a task-management SaaS FAQ assistant. All commands, library versions, and code snippets are tested and reproducible. The same structure applies to customer-support bots, documentation assistants, or any RAG application where correctness directly affects user trust.

Core Metrics Every RAG System Must Track

RAGAS provides four foundational metrics. Context precision counts how many of the retrieved chunks were relevant to the question according to an LLM judge. The formula is the sum of relevance indicators divided by the number of retrieved chunks. Context recall measures the fraction of ground-truth relevant sentences that appear in the retrieved set. Faithfulness decomposes the generated answer into atomic statements and verifies whether each statement is entailed by the retrieved context. Answer relevancy embeds the question and the generated answer then computes cosine similarity between them after removing irrelevant sentences.

DeepEval’s G-Eval extends this by accepting a free-form criterion such as “the answer must list exact button labels present in the current product version.” The judge produces a score from 0 to 1 plus a readable justification. This flexibility captures product-specific requirements that generic metrics miss.

Production thresholds calibrated on 120 human-rated examples for the task-management domain are: context precision greater than or equal to 0.85, context recall greater than or equal to 0.80, faithfulness greater than or equal to 0.90, and answer relevancy greater than or equal to 0.85. When any metric falls below its threshold for three consecutive nightly runs, an automated alert fires and the offending prompt or retrieval configuration is flagged for review.

Common failure patterns caught by these metrics include retrieval of outdated documentation (low context recall), the model inventing workflow steps not present in any chunk (low faithfulness), and the answer beginning with a friendly greeting instead of directly addressing the question (low answer relevancy). Each pattern maps to a distinct remediation: update the document index, tighten the system prompt, or add output format instructions.

Project Setup with Exact Versions

Begin with a clean Python 3.11 environment. The exact combination below avoids known compatibility issues between RAGAS callbacks and newer LangChain releases.

python -m venv venv
source venv/bin/activate
pip install ragas==0.1.21 deepeval==0.1.8 langsmith==0.1.125 langchain==0.3.0 langchain-openai==0.2.0 langchain-community==0.3.0 datasets==2.21.0 chromadb==0.5.5

After installation, create a .env file containing:

OPENAI_API_KEY=sk-proj-...
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=lsv2_pt_...
LANGCHAIN_PROJECT=taskmanager-rag-eval

Load the variables with dotenv or export them directly. Verify the setup by running a single RAGAS evaluation on two hand-crafted examples; the run should complete in under 45 seconds and return scores without traceback errors related to callback handlers. This step confirms that tracing, embeddings, and judges are wired correctly before scaling to larger datasets.

Building a Minimal RAG Application for the Example

The example FAQ assistant indexes seven markdown documents describing task creation, assignment, dependencies, notifications, and mobile sync behavior. Documents are split into 800-token chunks with 100-token overlap to preserve context across boundaries.

from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_core.documents import Document

loader = DirectoryLoader("faq_docs", glob="*.md")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", " ", ""]
)
chunks = splitter.split_documents(docs)

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}
)

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)

template = """Answer strictly using the provided context. 
If the context lacks sufficient information, state "I cannot answer this from the available documentation."
Cite the source chunk number in brackets after each factual sentence.

Context:
{context}

Question: {question}
Answer:"""

prompt = ChatPromptTemplate.from_template(template)

chain = (
    {"context": retriever | (lambda docs: "\n\n".join(doc.page_content for doc in docs)),
     "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

def answer_with_context(question: str) -> dict:
    retrieved_docs = retriever.invoke(question)
    answer = chain.invoke(question)
    return {
        "question": question,
        "answer": answer,
        "contexts": [doc.page_content for doc in retrieved_docs],
        "ground_truth": None  # populated later by human expert
    }

The answer_with_context helper returns the exact dictionary shape expected by both evaluation libraries. Running answer_with_context("How do I set a task dependency?") returns the generated answer plus the four retrieved chunks used, allowing immediate inspection of whether the correct policy document was surfaced.

Creating a Reusable Evaluation Dataset

A robust test set mixes simple lookup questions, multi-hop reasoning, negation handling, and deliberate out-of-scope queries. The dataset below contains eight concrete examples; expand it to 80–150 entries for statistical confidence.

from datasets import Dataset, DatasetDict

questions = [
    "How do I assign a task to a teammate in the web app?",
    "Can I create a dependency between two tasks?",
    "What notification is sent when a task becomes overdue?",
    "Does the mobile app support offline task creation?",
    "How do I bulk update task statuses?",
    "Is there a way to export completed tasks as CSV?",
    "What happens if I delete a project that contains open tasks?",
    "How are task comments ordered in the activity feed?"
]

# Generated answers and retrieved contexts would be added by running the RAG chain
# Ground truth written by product expert
ground_truths = [
    "Click the task, open the assignee dropdown in the right sidebar, and select a teammate.",
    "Open task A, click the dependencies button, search for task B, and choose blocking or blocked-by.",
    "An email and in-app notification are sent to the assignee and project owner.",
    "Yes, tasks created offline sync when connectivity returns.",
    "Select multiple tasks with shift-click then choose new status from the toolbar.",
    "Open the project, click Export, select Completed tasks, and choose CSV format.",
    "The project cannot be deleted until all open tasks are moved or completed.",
    "Newest comments appear at the bottom; the feed can be reversed with a toggle."
]

data = {
    "question": questions,
    "answer": [""] * len(questions),  # populated by RAG
    "contexts": [[] for _ in questions],
    "ground_truth": ground_truths
}

eval_dataset = Dataset.from_dict(data)
eval_dataset.save_to_disk("eval_dataset_v1")

After the first run of the RAG chain against these questions, replace the empty answer and contexts fields with real values, then manually verify and refine the ground_truth entries. Commit the finished dataset to git so every code change is scored against identical inputs.

Running RAGAS Evaluations

With the dataset prepared, execute the evaluation in one call. RAGAS automatically selects an LLM judge (default GPT-4o) and embeddings model.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

result = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=ChatOpenAI(model="gpt-4o", temperature=0),
    embeddings=OpenAIEmbeddings(model="text-embedding-3-small"),
    raise_exceptions=False
)

print(result)
print("Individual scores per row:")
for i, row in enumerate(result["faithfulness"]):
    print(f"Q{i}: faithfulness={row:.3f}")

The library returns both an aggregate score and per-row values. Store the full JSON output alongside the dataset version for historical comparison. If any single example scores below 0.6 on faithfulness, inspect the explanation trace that RAGAS can return when return_outputs=True is added; it lists which statements were unsupported and which context chunk should have prevented the hallucination.

Adding DeepEval G-Eval for Custom Criteria

DeepEval allows product-specific rubrics that RAGAS metrics alone cannot express. For the task manager we define correctness against exact UI labels and workflow steps.

from deepeval import evaluate
from deepeval.metrics import GEval, FaithfulnessMetric
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria="The response must reference actual button labels, menu paths, and behavior present in version 2.4 of the product. It must never invent non-existent features.",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
        LLMTestCaseParams.RETRIEVAL_CONTEXT
    ],
    threshold=0.85,
    model="gpt-4o",
    include_reason=True
)

test_cases = []
for row in eval_dataset:
    test_case = LLMTestCase(
        input=row["question"],
        actual_output=row["answer"],
        expected_output=row["ground_truth"],
        retrieval_context=row["contexts"]
    )
    test_cases.append(test_case)

results = evaluate(test_cases, [correctness_metric, FaithfulnessMetric(threshold=0.9)])
print(results)

Each failed test case returns a reason string such as “The answer mentions a ‘Bulk Actions’ toolbar that does not exist; the correct label is ‘Batch Operations’ in the current version.” These explanations accelerate prompt fixes. Run both RAGAS and DeepEval in the same script and combine scores with a weighted average (0.4 faithfulness, 0.3 correctness, 0.15 context recall, 0.15 answer relevancy) to produce a single composite quality score used by the CI gate.

Tracing Experiments with LangSmith

LangSmith records every retrieval call, prompt template, LLM completion, and evaluation score inside a shared project. After each evaluation run, upload the results so prompt variants can be compared.

from langsmith import Client
from langsmith.evaluation import LangChainStringEvaluator

client = Client()
dataset_name = "TaskManager-Eval-v1"

# Create or reuse dataset in LangSmith UI first
# Then run experiments
experiment_results = client.run_on_dataset(
    dataset_name=dataset_name,
    llm_or_chain_factory=lambda: chain,
    evaluation=LangChainStringEvaluator("qa", criteria="correctness"),
    project_name="prompt-v2-vs-v3-comparison",
    metadata={"prompt_version": "v3", "model": "gpt-4o-mini"}
)

Inside the LangSmith UI, filter experiments by metadata, view side-by-side metric deltas, and export CSV for further analysis. This workflow makes prompt engineering a measurable engineering discipline instead of trial-and-error.

Automating Evaluation in GitHub Actions

The CI workflow runs on changes to either the retrieval code, prompt templates, or the evaluation dataset itself. It fails the pull request if the composite score drops more than 0.05 or any single metric falls below its hard threshold.

name: LLM Evaluation Gate
on:
  pull_request:
    paths:
      - 'rag/**'
      - 'eval_dataset/**'
      - 'evaluation/**'

jobs:
  ragas-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      - run: pip install -r evaluation/requirements.txt
      - name: Execute evaluation suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGCHAIN_API_KEY: ${{ secrets.LANGCHAIN_API_KEY }}
        run: python evaluation/run_eval.py
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: evaluation/results.json

The run_eval.py script loads the dataset, runs both RAGAS and DeepEval, writes a JSON report, and exits with non-zero status if thresholds are violated. Nightly cron jobs run the full suite against 120 examples and post a trend chart to the team’s internal wiki.

Production Monitoring Strategy

Static datasets alone miss distribution shift. Implement asynchronous evaluation on live traffic by sampling 1.5 % of production conversations after stripping personally identifiable information. A lightweight FastAPI service receives LangSmith trace IDs, replays the question through the current chain, and records fresh metric scores in a TimescaleDB table.

A weekly SQL query computes rolling seven-day averages per metric. When faithfulness drops below 0.88, an automated Jira ticket is created containing the ten lowest-scoring examples with their judge explanations. This process caught a regression when a new document chunking strategy introduced overlapping policy text that confused the generator; the context precision metric fell from 0.89 to 0.71 within 48 hours of deployment.

Retention policy keeps raw traces for 30 days and aggregated scores indefinitely to support long-term trend analysis and model comparison after future LLM upgrades.

Choosing Between Local and Cloud Judges

Cloud judges using GPT-4o deliver the highest correlation with human raters but incur per-evaluation cost. Local judges running Llama-3.1-70B via vLLM reduce cost to zero after initial hardware investment yet require careful temperature tuning (0.0) and slightly modified rubrics because the model is less adept at following complex scoring instructions.

A hybrid approach works best: use the cloud judge for nightly full-suite runs and CI on critical paths, and the local judge for rapid local iteration during prompt development. Switching the LLM argument in both RAGAS and DeepEval requires only a LangChain wrapper around the vLLM OpenAI-compatible endpoint. Measure agreement between the two judges on a 40-example hold-out set; target Cohen’s kappa above 0.72 before trusting the local judge for gating decisions.

Cost and Latency Trade-offs

One complete RAGAS pass over eight examples with GPT-4o as judge costs roughly $0.09 and completes in 90 seconds on a standard laptop. Switching the judge to gpt-4o-mini lowers cost to $0.014 while preserving greater than 0.8 correlation on faithfulness for this domain. DeepEval G-Eval calls are 2.3× more expensive than RAGAS faithfulness because of longer chain-of-thought output tokens.

Latency impact is mitigated by running evaluations asynchronously after the user receives an answer. Only in CI is synchronous blocking evaluation acceptable. Caching identical (question, context, answer) triples reduces repeated LLM judge calls by approximately 40 % on iterative prompt work.

Trade-off summary: prioritize faithfulness and correctness over context precision during early development because generation quality matters more to users than perfect retrieval. Once faithfulness exceeds 0.92, shift focus to lifting context recall through embedding model upgrades or query rewriting.

Iterating on Retrieval Using Evaluation Signals

Low context recall (<0.75) indicates the retriever misses relevant information. Concrete fixes applied in sequence:

  1. Replace pure similarity search with MMR (maximal marginal relevance) using search_type="mmr", search_kwargs={"k": 6, "fetch_k": 20, "lambda_mult": 0.7}. This reduced redundant chunks and raised recall by 0.11.
  2. Implement HyDE: generate a hypothetical answer with a cheap model then embed that hypothetical for retrieval. Added 0.08 to context recall on multi-hop questions.
  3. Upgrade to voyage-law-2 embeddings for the legal-adjacent policy documents present in the FAQ. The domain-specific model delivered another 0.06 lift.
  4. Add metadata filtering so only documents whose version tag matches the current product version are considered.

After each change the identical evaluation dataset is re-run and the delta in every metric is recorded in a Markdown table inside the pull request description. This practice makes retrieval improvements visible and reproducible.

Prompt Engineering Driven by Low Faithfulness Scores

Repeated low faithfulness scores pointed to the model supplementing context with pre-training knowledge. The revised system prompt that lifted faithfulness from 0.74 to 0.95 across the test set reads:

You are a strict FAQ assistant. Follow these rules without exception:
1. Use ONLY information present in the provided context.
2. If the context does not contain the answer, respond exclusively with: "I cannot answer this from the available documentation."
3. After each factual sentence add a citation in the form [Chunk 3].
4. Never apologize or add meta commentary.

Context:
{context}

Question: {question}
Answer:

The citation requirement allows a post-processing script to parse bracketed numbers and verify they correspond to actually retrieved chunks, adding a deterministic check alongside the LLM judge. This combination of stronger instructions and verifiable output format eliminated the majority of hallucination failures.

Building a Custom Dashboard for Evaluation Trends

A lightweight Streamlit dashboard pulls results from the PostgreSQL table populated by the production monitoring service. It displays rolling seven-day averages for each metric, a histogram of per-example scores, and the five worst-performing questions with judge explanations. The dashboard also shows A/B test results when two prompt versions run in parallel on live traffic. Engineers consult the dashboard before and after every prompt or retrieval change to confirm expected movement in the composite score.

Alerting rules configured in the same database trigger PagerDuty when the 24-hour faithfulness average drops below 0.85. The combination of real-time monitoring, historical trends, and automatic ticketing closes the loop between production behavior and development priorities.

Scaling the Evaluation System

Once stable, extend the pipeline in these directions. First, add multilingual support by inserting an automatic translation step before scoring non-English questions and comparing against English ground truth. Second, maintain a model comparison matrix that runs the identical test set against gpt-4o, Claude-3-5-Sonnet, and local Llama-3.1-70B, ranking each by weighted composite score and token cost. Third, route the lowest-scoring 5 % of examples to a human review queue where raters refine ground truth or adjust rubric wording. Fourth, add regression tests that assert exact string matches on 12 critical policy questions; any deviation fails the build even if numeric metrics pass.

These layers convert the evaluation system from a development aid into a comprehensive quality gate that scales with the complexity of the LLM feature set while maintaining deterministic release criteria.

The full repository—including documents, evaluation scripts, GitHub Actions configuration, Streamlit dashboard, and example dataset—is linked in the site resources. Replace the FAQ documents with your own content, run the initial evaluation, tune thresholds to match your manual spot checks, and the automated feedback loop will drive continuous improvement without guesswork.

Keep reading

Related Articles