Query Rewriting

Query rewriting is the family of techniques that transform a user’s raw query into a different form before it ever reaches the retriever, rather than embedding or keyword-matching it as-is. The motivation is simple: the way a person asks a question rarely matches the vocabulary, structure, or scope of the documents that actually contain the answer. A short, ambiguous, or conversational query embeds into a different region of vector space than the long, formal passage that answers it, and the same mismatch trips up keyword search too. Rather than trying to fix this on the retriever’s side, query rewriting fixes it upstream, by changing the query itself. It sits in the same neighborhood as HyDE and Corrective RAG, both of which also intervene before or around retrieval, but rewriting is the broadest family: it covers everything from a single LLM-prompted reformulation to a trained model optimized end-to-end against retrieval quality.

The Vocabulary Mismatch Problem

Three distinct failure patterns motivate query rewriting, and each has produced its own technique:

• Underspecification. A follow-up message in a chat thread (“what about the enterprise plan?”) is meaningless to a retriever without the prior turns. It needs to be rewritten into a self-contained query before search even makes sense.
• Vocabulary gap. A user’s phrasing (“my app keeps crashing on startup”) shares little surface vocabulary with the documentation that would fix it (“resolving initialization failures in the runtime”). A single rewrite might miss the right phrasing; several attempts, each framed differently, cover more ground.
• Scope mismatch. A specific, narrow question (“what was Acme Corp’s Q3 2024 revenue?”) sometimes retrieves better when it’s first answered by pulling in the broader context around it (“what is Acme Corp’s recent financial performance?”), especially when the narrow fact is easy to look up wrong but the broader context grounds the model in what’s actually true.

No single rewrite strategy fixes all three, which is why query rewriting isn’t one technique but a small toolbox, each entry aimed at a different mismatch.

Four Ways to Rewrite a Query

graph LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    Q([Raw User Query]):::data --> RW[Rewriter:<br/>LLM prompt or trained model]:::process
    RW -->|Single reformulation| S1[Rewrite-Retrieve-Read]:::process
    RW -->|N parallel variants| S2[Multi-query fan-out<br/>+ RRF fusion]:::process
    RW -->|Higher-level abstraction| S3[Step-back prompting]:::process
    RW -->|Sub-questions| S4[Query decomposition]:::process
    S1 --> R[Retriever]:::process
    S2 --> R
    S3 --> R
    S4 --> R
    R --> G[Generation]:::output

All four share the same position in the pipeline, between the user’s input and the retriever, but differ in what they output: one clarified query, several parallel variants, one more abstract query, or several narrower sub-questions. They also differ sharply in cost: a single reformulation adds one LLM call and no extra retrieval calls, fan-out multiplies retrieval calls by the number of variants generated, and decomposition adds both an LLM call and a retrieval call per hop. None of that cost is optional overhead to eliminate, it’s the mechanism itself: fan-out only helps because it retrieves multiple times, decomposition only helps because it reasons in multiple steps. Picking a rewriting strategy is really picking how much of that added latency and spend a given query type is worth.

Rewrite-Retrieve-Read: A Trained Rewriter, Not Just a Prompt

The technique that gave this family its name comes from Ma et al.’s 2023 paper, which reframes the standard “retrieve-then-read” pipeline as “rewrite-retrieve-read.” The simplest version of this just prompts an LLM to reformulate the query, and that alone helps. The paper’s contribution goes further: it trains a small T5-large model as a dedicated rewriter, sitting in front of a frozen, black-box LLM reader (the paper tests both ChatGPT and Vicuna-13B), so the rewriter can be optimized without ever touching the reader.

Training happens in two stages. First, a warm-up phase distills rewriting behavior from an LLM’s own pseudo-rewrites into the small T5 model via supervised fine-tuning, giving it a reasonable starting policy. Second, that policy is refined with reinforcement learning: Proximal Policy Optimization (PPO), where the reward comes directly from the frozen reader’s downstream task performance (exact match, F1, or retrieval hit rate), with KL regularization keeping the rewriter from drifting into degenerate, unnatural queries just to game the reward. The result is a rewriter trained not to produce a query that looks better, but one that measurably improves what the reader can do with it, evaluated on open-domain QA benchmarks including HotpotQA and AmbigNQ.

# Scenario: swapping a prompted rewrite for a small trained rewriter that's
# been optimized specifically against this reader's downstream accuracy
class TrainableRewriter:
    def __init__(self, model_path: str):
        self.model = load_t5_model(model_path)  # fine-tuned via warm-up + PPO

    def rewrite(self, query: str) -> str:
        return self.model.generate(f"rewrite: {query}")

def rewrite_retrieve_read(query, rewriter, retriever, reader):
    rewritten = rewriter.rewrite(query)          # trained, not just prompted
    docs = retriever.search(rewritten, top_k=5)
    return reader.answer(query, docs)            # original query still goes to the reader

The practical reason to train a rewriter instead of just prompting one: a general-purpose LLM prompted to “rewrite this query” optimizes for a plausible-looking rewrite, not necessarily one that retrieves better documents for this specific reader and corpus. A rewriter trained against the reader’s own feedback closes that gap, at the cost of needing a training pipeline most teams skip in favor of prompting.

Multi-Query Fan-Out and Fusion (RAG-Fusion)

Instead of producing one better query, this approach produces several different ones and retrieves for all of them in parallel. An LLM generates a handful of variants of the original query, each framed from a different angle, each is retrieved independently, and the resulting ranked lists are merged with Reciprocal Rank Fusion (RRF). This is the approach popularized as RAG-Fusion by Rackauckas, built on the same fusion primitive covered in hybrid search.

# Scenario: a single narrow phrasing might miss documents indexed under
# different terminology, so generate several angles and fuse the results
def generate_query_variants(query: str, llm, n: int = 4) -> list[str]:
    prompt = f"Generate {n} different search queries that explore this question from different angles:\n{query}"
    return llm.complete(prompt).strip().split("\n")[:n]

def multi_query_retrieve(query, retriever, llm, n=4, k=60):
    variants = generate_query_variants(query, llm, n)
    ranked_lists = [
        [doc.id for doc in retriever.search(v, top_k=20)] for v in variants
    ]
    scores = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

The trade-off is direct: more variants catch more of the corpus’s terminology at the cost of proportionally more retrieval calls and one extra LLM call to generate them. The widget below models that trade-off with a simplified assumption, that each query variant has some independent probability of retrieving the target document, so fusing more variants compounds toward near-certain recall with diminishing returns.

Step-Back Prompting: Abstracting Before Retrieving

Where multi-query fan-out widens a search, step-back prompting, introduced by Zheng et al. in 2023, goes up a level of abstraction instead. Given a specific question, the LLM first generates a broader “step-back” question, retrieves grounding for both the original and the step-back question, and only then answers. The intuition: a narrow factual question is easy to answer wrong from a single closely matching but subtly incorrect passage, while the broader question surfaces the general context needed to sanity-check the specific answer.

# Scenario: "What was Acme Corp's Q3 2024 revenue?" is easy to answer wrong
# from one stale press release; the step-back question grounds the model
# in broader context first
def step_back_answer(query, llm, retriever):
    step_back_prompt = f"What is a more general question that would help answer: {query}"
    step_back_query = llm.complete(step_back_prompt).strip()
    # step_back_query: "What has Acme Corp's recent financial performance looked like?"

    specific_docs = retriever.search(query, top_k=3)
    general_docs = retriever.search(step_back_query, top_k=3)

    context = specific_docs + general_docs
    return llm.complete(f"Context:\n{context}\n\nQuestion: {query}")

The paper’s own numbers make the case for retrieval-augmented step-back concretely on TimeQA, a benchmark of time-sensitive factual questions: combined with standard retrieval augmentation, step-back prompting reached 68.7% accuracy, against baselines of 45.6% for GPT-4 and 41.5% for PaLM-2L answering the same questions without the step-back reformulation. It corrected 39.9% of the baseline’s wrong answers while introducing new errors on only 5.6% of previously correct ones, a favorable trade specifically because the step-back question tends to retrieve stable, general context rather than a single time-sensitive fact that might be wrong or outdated.

Query Decomposition for Multi-Hop Questions

The fourth pattern goes the opposite direction from step-back: instead of one broader question, it breaks a complex question into several narrower ones. A question like “was the director of the highest-grossing film of 2023 also behind an Oscar winner?” can’t be answered by any single retrieval, it requires first identifying the film, then its director, then that director’s other work. Decomposition splits this into an ordered chain of sub-questions, retrieving and answering each in turn, feeding each answer into the next sub-question.

# Scenario: a multi-hop question where no single retrieval call contains
# the full answer, it has to be assembled hop by hop
def decompose_and_answer(query, llm, retriever, reader, max_hops=3):
    sub_questions = llm.complete(
        f"Break this question into an ordered list of simpler sub-questions:\n{query}"
    ).strip().split("\n")[:max_hops]

    context_so_far = []
    for sub_q in sub_questions:
        docs = retriever.search(sub_q, top_k=3)
        answer = reader.answer(sub_q, docs + context_so_far)
        context_so_far.append(f"{sub_q} -> {answer}")

    return reader.answer(query, context_so_far)

This is the most expensive rewriting pattern, one LLM call per hop plus a retrieval per hop, but it’s the only one of the four that can answer questions no single retrieval, however well-phrased, could ever satisfy on its own. The ordering matters too: each sub-question is generated and retrieved only after the previous hop’s answer is known, rather than all sub-questions being planned up front, since the second hop in the example above (finding the director) depends on the first hop’s answer (which film), not on the original question directly. Systems that instead decompose everything up front, before any retrieval happens, tend to fail on questions where a later sub-question can’t even be phrased correctly until an earlier one has been answered.

Query Rewriting vs. HyDE: Two Different Bridges

Both techniques sit between the query and the retriever and both address vocabulary mismatch, which makes them easy to conflate, but they intervene differently:

In practice the two compose rather than compete: a query can be rewritten for clarity first, then that clarified query can be the input to a HyDE-style hypothetical document generation step, each closing a different part of the gap between question and answer.

What’s New (2025-2026)

• Query rewriting as a standard agent tool call, not a fixed pipeline stage. Agentic RAG systems increasingly treat rewriting as one action an agent can choose to invoke, deciding at run time whether the raw query needs reformulation, fan-out, or decomposition, rather than always running the same fixed rewrite step.
• Rewriting folded into corrective pipelines. Corrective RAG’s web-search fallback is itself a small, purpose-built query rewrite, converting a natural-language question into a keyword-oriented search query, showing rewriting increasingly living inside other retrieval techniques rather than as a standalone stage.
• Framework-native support. LangChain’s MultiQueryRetriever and LlamaIndex’s step-decomposition and query-transform modules have made multi-query fan-out and step-back prompting default, low-code options rather than techniques teams implement from a paper.
• Reinforcement-learning-trained rewriters extending beyond QA. Follow-on work has applied the rewrite-retrieve-read reward-from-downstream-metric pattern to code search and structured retrieval settings, rewarding the rewriter against retrieval hit rate for domains beyond open-domain question answering.

Tuning and Practical Guidance

None of these four techniques is a strict upgrade over the others, they answer different questions about what’s wrong with the raw query: is it underspecified, oddly worded, too narrow, or too complex. Production RAG systems increasingly pick one per query type at run time rather than committing to a single rewriting strategy for every request.

# Scenario: a support chatbot where the user's second message is a
# follow-up ("what about the enterprise plan?") that means nothing to a
# retriever without the prior turn's context
def rewrite_followup(conversation: list[dict], llm) -> str:
    history = "\n".join(f"{m['role']}: {m['content']}" for m in conversation)
    prompt = (
        "Given this conversation, rewrite the final user message as a "
        "single, self-contained search query that would make sense with "
        "no prior context. Return only the rewritten query.\n\n"
        f"{history}"
    )
    return llm.complete(prompt).strip()

conversation = [
    {"role": "user", "content": "Does your product support SSO?"},
    {"role": "assistant", "content": "Yes, SAML and OIDC are supported on paid plans."},
    {"role": "user", "content": "What about the enterprise plan?"},
]

query = rewrite_followup(conversation, llm)
# query: "Does the enterprise plan support SSO (SAML and OIDC)?"
results = vector_store.search(query, top_k=5)