DeepEval · RAG evaluation · QA practice

Five metrics.
Three broken components.
One useful diagnosis.

A metric deserves a place in your suite only when its failure changes what an engineer does next. These five separate retrieval defects, generation defects, and product-policy defects without turning the dashboard into a catalog.

Verified against DeepEval 4.2.0 on 30 August 2026. The framework recommends no more than five metrics; this exact set is our support-RAG choice, not an official preset.

8 min read Failure-first Single-turn RAG

01 Diagnose first

A score is not a finding

“Quality dropped to 0.68” is not actionable. Did retrieval miss the policy? Did the re-ranker bury it? Did the model ignore it? Or did the answer violate a business rule that generic RAG metrics do not know?

The useful unit of evaluation is therefore not the metric. It is the metric → component → next action chain.

Do not ask which metric is most impressive. Ask which failure it isolates—and whether your team knows what to change when it turns red.

Observed pattern	Likely component	First engineering move
Recall falls	Retriever coverage	Inspect top-k, chunk boundaries, filters, and embeddings.
Recall holds; precision falls	Ranking	Inspect ordering, duplicate chunks, and the re-ranker.
Retriever metrics hold; faithfulness falls	Generator grounding	Inspect the prompt, context use, and model behavior.
Faithfulness holds; relevancy falls	Generator focus	Remove tangents, boilerplate, and unnecessary caveats.
Generic metrics hold; policy score falls	Product contract	Inspect the rubric, expected answer, and application logic.

Route the failure before tuning anything

The diagram is intentionally asymmetric: generic RAG metrics isolate plumbing; the custom metric checks the rule that makes your product yours.

02 The five metrics

Each metric owns one question

The first four form a diagnostic set for RAG. The fifth encodes the business requirement the generic set cannot infer. More metrics are justified only when they expose a new failure mode.

01 · Generator grounding

Faithfulness

Did the answer stay inside the retrieved evidence?

DeepEval extracts claims from the answer and checks whether each claim is supported by the retrieval context. A low result is a generator finding: the evidence that reached the model did not justify what it said.

supported claims ÷ total claims

A true statement can still be unfaithful when the retrieved context does not contain it. That is the point: this metric tests grounding, not world knowledge.

inputactual_outputretrieval_context

02 · Generator focus

Answer Relevancy

Did the answer address the user’s request?

This referenceless metric evaluates statements in the output against the input. It does not inspect retrieval context and does not establish factual correctness.

relevant answer statements ÷ total answer statements

Long preambles, unrelated caveats, and friendly-but-useless filler can reduce the score. Pair it with Faithfulness: one catches drift from the question; the other catches drift from the evidence.

inputactual_output

03 · Ranking

Contextual Precision

Did useful chunks appear before noise?

Contextual Precision judges each retrieved node for relevance and computes a weighted cumulative precision. Relevant nodes earn more credit when they appear near the top of the ranked context.

average of precision@k at positions where the retrieved node is relevant

High recall with low precision means the answer material exists, but ordering is weak. Inspect the re-ranker, duplicate chunks, filters, and top-k—not the answer prompt.

inputactual_outputexpected_outputretrieval_context

04 · Coverage

Contextual Recall

Did retrieval fetch enough for the ideal answer?

DeepEval breaks the expected output into statements and checks which statements can be attributed to the retrieval context. It deliberately evaluates retrieval against the ideal answer, not the answer your generator happened to produce.

expected-output statements attributable to context ÷ all expected-output statements

Low recall points toward coverage: top-k, chunk boundaries, metadata filters, or embeddings. The generator cannot produce a complete grounded answer from evidence it never received.

inputactual_outputexpected_outputretrieval_context

05 · Product contract

G-Eval

Did the answer obey the rule our business cares about?

G-Eval is the custom slot. Use it for correctness, completeness, tone, or policy behavior that the four generic metrics do not define. Give it explicit evaluation steps and only the test-case fields those steps need.

0–10 raw judge score, optionally probability-weighted, then normalized to 0–1

DeepEval recommends supplying stable evaluation_steps when you have them. If your criteria can be expressed as a rigid decision tree and repeatability matters more than nuance, investigate the DAG metric instead.

actual_outputexpected_output

Evaluating an agent? Replace slots; do not simply add more. Task Completion covers the trajectory, while Tool Correctness diagnoses tool selection. A five-slot budget still forces ownership.

03 One case, end to end

Ticket 7319

A customer received a cracked ceramic bottle. They ask whether it can be replaced and whether return shipping costs extra. One case is enough to see why five separate verdicts beat one blended “quality” score.

The golden case

Input: “My ceramic bottle arrived cracked. Can I get a replacement, and do I pay return shipping?”
Expected output: Report the damage within 14 days for a free replacement. The company covers return shipping.
Retrieved context: Chunk 1: damaged-item reports are accepted within 14 days. Chunk 2: replacement and return shipping are free. Chunk 3: unrelated loyalty-points policy.
Actual output: “Yes. Contact support within 30 days. We will replace it, but you may need to pay return shipping.”

test_ticket_7319.py

from deepeval.metrics import (
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    FaithfulnessMetric,
    GEval,
)
from deepeval.test_case import LLMTestCase, SingleTurnParams

case = LLMTestCase(
    input="My ceramic bottle arrived cracked...",
    actual_output="Contact support within 30 days...",
    expected_output="Report it within 14 days...",
    retrieval_context=[policy_a, policy_b, loyalty_policy],
)

policy_correctness = GEval(
    name="Damage policy correctness",
    evaluation_steps=[
        "Compare the deadline in actual and expected output.",
        "Check who pays return shipping.",
        "Fail answers that reverse either policy fact.",
    ],
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.EXPECTED_OUTPUT,
    ],
    threshold=0.8,
)

metrics = [
    FaithfulnessMetric(threshold=0.8),
    AnswerRelevancyMetric(threshold=0.7),
    ContextualPrecisionMetric(threshold=0.7),
    ContextualRecallMetric(threshold=0.7),
    policy_correctness,
]

Recall may pass.

The retrieved context contains both facts needed for the ideal answer.

Precision may be imperfect.

The loyalty chunk adds noise, but the useful chunks still rank ahead of it.

Relevancy may pass.

The response stays on the customer’s question. On-topic does not mean correct.

Faithfulness should fail.

The 30-day deadline and customer-paid shipping contradict the retrieved policy.

G-Eval should fail.

The product-specific rubric makes both policy reversals explicit and reviewable.

04 Thresholds

Calibrate a gate; do not invent one

DeepEval defaults most metric thresholds to 0.5, but a framework default is not your release policy. The example thresholds above are starting points for calibration, not universal standards.

Start with score-only observation

Use threshold=None while collecting a healthy baseline. Scores and reasons are recorded, but the metric has no pass/fail opinion.

Label failures before gating

Compare metric judgments with human QA labels. If the judge disagrees on cases that matter, repair the rubric, examples, or evaluation model first.

Set thresholds from the healthy band

Choose a boundary that catches meaningful regressions without making ordinary judge variance look like a production incident.

Use strict mode only for invariants

strict_mode=True requires perfection, makes the score binary, and sets the threshold to 1. Reserve it for rules that truly allow no partial credit.

Monitor known-noisy metrics

A metric marked flaky=True still produces its score, reason, and verdict, but its failure does not fail the test case.

Read reasons and traces

A score is a locator, not a confidence percentage. Review which claim, statement, or chunk produced the judgment before tuning the system.

05 Grounding

Sources

The article uses official DeepEval documentation and source metadata. Metric counts and defaults change; the component boundaries and required inputs below were checked on 30 August 2026.

DeepEval — Introduction to LLM Evaluation Metrics. Shared score direction, threshold behavior, strict mode, flaky metrics, score-only mode, and the current metric categories. deepeval.com/docs/metrics-introduction
DeepEval — Faithfulness. Required fields, grounding scope, calculation, and the distinction between generation and retrieval failure. deepeval.com/docs/metrics-faithfulness
DeepEval — Answer Relevancy. Referenceless inputs, statement-level relevance, and why factuality must be evaluated separately. deepeval.com/docs/metrics-answer-relevancy
DeepEval — Contextual Precision. Weighted cumulative precision, order sensitivity, required fields, and its role in diagnosing ranking. deepeval.com/docs/metrics-contextual-precision
DeepEval — Contextual Recall. Expected-output attribution, required fields, and its role in diagnosing retrieval coverage. deepeval.com/docs/metrics-contextual-recall
DeepEval — G-Eval. Evaluation steps, selected test-case parameters, score normalization, limitations, and the DAG alternative. deepeval.com/docs/metrics-llm-evals
Confident AI — DeepEval 4.2.0 release and source. Official release metadata for the higher-is-better score contract and the version used for this article. github.com/confident-ai/deepeval

Five DeepEval Metrics That Turn RAG Scores into Engineering Decisions

Building Applications with AI Agents

Prompting

Five metrics.
Three broken components.
One useful diagnosis.