Corrective RAG (CRAG)

Corrective RAG (CRAG) adds a checkpoint that standard retrieval-augmented generation doesn’t have: before any retrieved text reaches the LLM, a lightweight evaluator scores how well it actually answers the query, and that score decides what happens next. Vanilla RAG has no such gate. It hands the retriever’s top-k results straight to the generator and trusts, without checking, that they’re relevant. CRAG was introduced in a January 2024 paper by Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling as a way to make that trust conditional instead of automatic, and by 2025-2026 the pattern it popularized, evaluate before you generate, has become a standard node type in agentic RAG graphs built with LangGraph and comparable orchestration frameworks.

The Problem: RAG Trusts Its Retriever Blindly

A retriever’s top-k results are a similarity ranking, not a relevance guarantee. A stale document, a chunk that’s topically adjacent but doesn’t actually answer the question, or a passage about a different entity with the same name can all rank highly and still be wrong. Standard RAG concatenates whatever comes back into the context window regardless, and the LLM has no signal that anything is off. Two failure modes follow from this: the model either hallucinates around thin or irrelevant context, or worse, it confidently repeats something false because a low-relevance chunk happened to state it plainly. Neither failure is visible from the retrieval step alone; both only surface once the model has already generated an answer, which is too late to correct cheaply.

CRAG’s premise is that this failure is preventable earlier in the pipeline. If a system can tell, before generation, that its retrieved evidence is weak, it can react: clean up what it has, go get better evidence, or do both, instead of asking the LLM to paper over the gap.

How CRAG Works: Evaluate, Then Correct

graph TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    Q([User Query]):::data --> R[Retriever:<br/>vector store top-k]:::process
    R --> E{Retrieval Evaluator:<br/>score each document}:::process
    E -->|Correct: above upper threshold| REF[Decompose then recompose:<br/>keep only relevant strips]:::process
    E -->|Ambiguous: between thresholds| BOTH[Refine internal docs<br/>and search the web]:::process
    E -->|Incorrect: below lower threshold| WEB[Discard internal docs:<br/>rewrite query, search the web]:::process
    REF --> G[Generation]:::output
    BOTH --> G
    WEB --> G

The evaluator runs once per retrieved document (in the paper’s implementation, once on the top-ranked document to decide the overall action). Three outcomes are possible:

• Correct. The retriever did its job. The documents go through a refinement step, decompose then recompose, to strip out irrelevant sentences before they reach the LLM.
• Incorrect. The retriever’s results are unusable. They’re discarded entirely, the original query is rewritten into a search-engine-friendly form, and a live web search replaces internal retrieval as the sole evidence source.
• Ambiguous. The evaluator isn’t confident either way. Rather than guess, CRAG hedges: it refines the internal documents and runs a web search, then combines both into the final context.

This three-way split, rather than a binary keep/discard decision, is what makes CRAG distinct from a simple relevance filter. It gives the system a middle path for the case that’s most common in practice: results that are neither clearly right nor clearly wrong.

The Retrieval Evaluator and Its Thresholds

The paper’s evaluator is deliberately small: a fine-tuned T5-large (0.77B parameters), trained on roughly 12,600 labeled query-document pairs derived from PopQA, with positive matches labeled 1 and negative matches labeled -1. At inference time it outputs a continuous confidence score between -1 and 1 for a given query-document pair, and that score is compared against an upper and lower threshold to pick the action. Critically, those thresholds aren’t fixed constants, the paper tunes them per dataset:

Notice how permissive the lower thresholds are, PopQA’s -0.99 means a document has to score almost the worst possible before CRAG discards it outright and trusts a web search alone. In practice this means the paper’s default configuration rarely fires the pure “Incorrect” path; most weak retrievals land in “Ambiguous” and get the hedge treatment instead. This is a real tuning decision, not an incidental detail, and it’s exactly the kind of knob a team adopting CRAG has to set for its own corpus and error tolerance.

Production implementations frequently swap the trained T5 evaluator for a general-purpose LLM call (a small model like GPT-4o-mini, prompted for a structured relevance judgment) instead of collecting labeled data and fine-tuning a dedicated scorer. That trades a training and hosting cost for a per-query inference cost, and it’s the approach most LangGraph-based CRAG tutorials use.

Decompose Then Recompose: Refining What the Retriever Got Right

Scoring “Correct” doesn’t mean a document is perfect, it usually still contains sentences that don’t help. A Wikipedia infobox page might carry the right answer buried under unrelated trivia; a support article might mix the relevant fix in with boilerplate. CRAG’s refinement step handles this with a decompose-then-recompose algorithm:

1. Decompose. Split each retrieved document into “knowledge strips”: a single sentence for short passages, or a small multi-sentence chunk for longer documents.
2. Score each strip. Run the same evaluator on every strip independently against the query.
3. Filter. Discard strips scoring below a fixed threshold (-0.5 in the paper), regardless of how well the document scored as a whole.
4. Recompose. Stitch the surviving strips back together in their original order into one refined context block.

# Scenario: a document scored "Correct" overall, but still needs its
# boilerplate and off-topic sentences stripped before it reaches the LLM
STRIP_KEEP_THRESHOLD = -0.5

def decompose_then_recompose(scored_docs, evaluator, query):
    refined_blocks = []
    for doc in scored_docs:
        strips = split_into_strips(doc.text)              # sentence or small chunk granularity
        kept = [s for s in strips if evaluator.score(query, s) >= STRIP_KEEP_THRESHOLD]
        if kept:
            refined_blocks.append(" ".join(kept))          # recompose in original order
    return "\n\n".join(refined_blocks)

def split_into_strips(text: str, max_words_per_strip: int = 40) -> list[str]:
    sentences = text.split(". ")
    strips, current = [], []
    for sentence in sentences:
        current.append(sentence)
        if sum(len(s.split()) for s in current) >= max_words_per_strip:
            strips.append(". ".join(current))
            current = []
    if current:
        strips.append(". ".join(current))
    return strips

This is a cheap, model-agnostic filter, no generation happens here, just repeated scoring calls against the same evaluator already used for the document-level decision. Its benefit compounds with context-window economics: a shorter, denser context block leaves more room for the LLM’s actual reasoning and reduces the chance that an irrelevant sentence gets picked up and repeated in the answer.

The Web Search Fallback

When the evaluator triggers “Incorrect” or “Ambiguous,” CRAG doesn’t reuse the user’s raw query for the web search, it rewrites it first into a keyword-oriented search query (the original paper uses ChatGPT for this step), issues it against a search API, and, per the paper, preferentially selects Wikipedia among the results as a higher-trust source when available. Retrieved web content is passed through the same decompose-then-recompose refinement as internal documents before being added to context. This is one of the clearest points of overlap with query rewriting: CRAG’s web-search branch is itself a small, purpose-built query rewriting step, optimized for a search engine rather than for a vector index.

# Scenario: the query "who's on the on-call rotation this week" needs to
# become a keyword query before it's useful against a general web search API
def rewrite_for_search(query: str, llm) -> str:
    prompt = f"Rewrite this question as a short, keyword-focused web search query:\n{query}"
    return llm.complete(prompt).strip()

CRAG vs. Self-RAG vs. Plain RAG

The paper’s own experiments run CRAG both standalone and layered on top of Self-RAG, and it improves results in both configurations, which is the strongest evidence that the two techniques address different parts of the problem: Self-RAG teaches the generator to reflect on what it produces, CRAG checks the evidence before generation even starts.

Results on the Original Benchmarks

PubHealth’s outsized jump makes intuitive sense: fact verification is exactly the task where a single wrong-but-plausible retrieved document does the most damage, since the model’s entire answer hinges on one true/false judgment. Catching and correcting that document before generation has an outsized effect precisely because there’s no room downstream to average the error away.

What’s New (2025-2026)

• CRAG as a graph node, not a fixed pipeline stage. LangGraph and comparable agentic frameworks implement the evaluate-then-branch logic as a conditional edge in a broader agent graph, often invoked as one tool among several rather than a mandatory step every query passes through.
• LLM-as-judge evaluators over trained scorers. Most production implementations skip fine-tuning a dedicated T5 evaluator and instead use a structured-output call to a small LLM (commonly GPT-4o-mini class models) to produce the relevance judgment, trading a hosting and training cost for a per-query inference cost.
• Simplified refinement in practice. Several widely used tutorials and reference implementations skip the full decompose-then-recompose step for a first pass, judging that the branching logic (correct, ambiguous, incorrect) captures most of the benefit on its own, and adding strip-level filtering only once the coarser routing is in place.
• Paired with hybrid retrieval upstream. CRAG is increasingly stacked after hybrid search and a reranker rather than a single vector lookup, so the evaluator is grading an already-fused, already-reranked candidate list instead of raw top-k similarity results.

When to Use CRAG

CRAG doesn’t replace a good retriever, hybrid search, or a reranker; it assumes all three already exist and adds the one thing none of them provide on their own: an explicit, checkable decision about whether to trust what came back before committing it to the model’s context.

# Scenario: an internal support bot deciding whether to trust its vector
# store or fall back to a live web search, based on evaluator confidence
from dataclasses import dataclass

UPPER_THRESHOLD = 0.59   # above this: Correct, per the paper's PopQA setting
LOWER_THRESHOLD = -0.99  # below this: Incorrect, per the paper's PopQA setting

@dataclass
class ScoredDoc:
    text: str
    score: float  # evaluator output, -1 (irrelevant) to 1 (highly relevant)

def evaluate(query: str, docs: list[str], evaluator) -> list[ScoredDoc]:
    # evaluator is any model (a trained T5, or an LLM structured-output call)
    # that returns a relevance score for a (query, document) pair
    return [ScoredDoc(d, evaluator.score(query, d)) for d in docs]

def corrective_retrieve(query, vector_store, web_search, evaluator):
    candidates = evaluate(query, vector_store.search(query, top_k=5), evaluator)
    best = max(candidates, key=lambda d: d.score)

    if best.score > UPPER_THRESHOLD:
        action = "correct"      # trust the retriever, just clean up the docs
        context = decompose_then_recompose(candidates, evaluator, query)
    elif best.score < LOWER_THRESHOLD:
        action = "incorrect"    # don't trust the retriever at all
        context = web_search.run(rewrite_for_search(query))
    else:
        action = "ambiguous"    # hedge: use both sources
        context = (
            decompose_then_recompose(candidates, evaluator, query)
            + web_search.run(rewrite_for_search(query))
        )

    return action, context