Everything in Modules 3 and 4 assumed a reasonable, well-formed query showed up at the front door of the retrieval system, ready to be embedded and searched. Production traffic disagrees with that assumption constantly. Real queries look like “what about contractors?”, “compare ours to Google’s”, “that thing Sarah presented about churn last quarter, remind me what the numbers were.” The best index in the world, tuned exactly as the previous four lessons describe, can’t rescue a query that doesn’t contain its own meaning in a form retrieval can act on. Query intelligence is the thin, fast layer sitting in front of everything else that fixes the query before retrieval spends any real work on it.
Rewriting: every chat turn after the first
In any genuinely conversational product, most queries past the very first turn are structurally incomplete on their own: pronouns pointing backward at something mentioned earlier, ellipsis like “and for the UK?” that only makes sense with prior context, typos accumulated from fast typing, and internal company shorthand that assumes shared context the retrieval system doesn’t have. The fix is mechanical, well-understood, and cheap to run: a small, fast model rewrites the current turn into a standalone query, conditioned explicitly on recent conversation history.
# Query rewrite: small model, strict contract, ~150ms.
# This runs on EVERY conversational turn, so speed matters
# more than brilliance; a haiku-class model is plenty.
REWRITE = """Given the conversation and the user's latest message,
rewrite the latest message as ONE standalone search query that
contains all context needed to understand it. Keep the user's
intent exactly. Expand pronouns and references. Fix obvious typos.
Return only the query text."""
async def rewrite_query(history: list[Turn], latest: str) -> str:
recent = format_turns(history[-6:]) # last 3 exchanges suffice
resp = await small_llm(
system=REWRITE,
user=f"Conversation:\n{recent}\n\nLatest message: {latest}",
max_tokens=80,
)
return resp.strip() or latest # empty rewrite -> fall back to rawTwo operational notes worth internalizing, since both come up repeatedly once this ships to real users. First, keep the raw, original query around alongside the rewritten one, don’t discard it. The rewritten query is what drives retrieval, since it’s optimized for that purpose, but the original wording goes into the generation prompt, so the final answer addresses what the user actually, literally said rather than the internal reformulation only your pipeline ever saw. Second, this stage fails open, following the exact pattern established back in Lesson 2’s safe() wrapper: if the rewrite call times out or errors, search the raw query as-is rather than blocking the request. A somewhat worse search still beats no answer at all.
Decomposition: questions that are secretly several questions
Some questions can’t be fixed by rewriting alone, because they aren’t incomplete, they’re genuinely compound. “How does our parental leave compare to Google’s?” or “Which of our EU customers are affected by the pricing change announced in March?” Both examples have their supporting evidence living in disjoint parts of the corpus, and a single averaged query vector lands somewhere unhelpful in the middle, retrieving neither piece particularly well.
The pattern here is decompose first, retrieve per sub-query independently, then synthesize the results together during generation:
flowchart TD
Q["'How does our parental leave\ncompare to Google's?'"] --> D["Decomposer\n(one small-LLM call)"]
D --> S1["Sub-query A:\n'company parental leave\npolicy duration'"]
D --> S2["Sub-query B:\n'Google parental leave\nbenchmark weeks'"]
S1 --> R1["Hybrid retrieve + rerank"]
S2 --> R2["Hybrid retrieve + rerank"]
R1 --> SYN["Generation synthesizes\nacross both evidence sets"]
R2 --> SYN
style Q fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style D fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style S1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style S2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style R1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style R2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style SYN fill:#fff7ed,stroke:#f59e0b,color:#0F172A# Rewrite AND decomposition, in ONE structured call. Doing this
# as a single round trip matters: two sequential LLM calls on
# the serving path double the latency this stage adds to every
# conversational turn.
from pydantic import BaseModel
class QueryPlan(BaseModel):
rewritten: str # standalone query for the simple case
needs_decomposition: bool
sub_queries: list[str] = [] # only populated when compound
PLAN_PROMPT = """Given the conversation and latest message, produce
a standalone rewritten query. If the question genuinely requires
evidence from more than one distinct topic to answer (a comparison,
a multi-part question), also set needs_decomposition=true and list
up to 3 independent sub-queries. Most questions do NOT need this."""
plan = await small_llm.parse(
system=PLAN_PROMPT,
user=f"Conversation:\n{recent}\n\nLatest: {latest}",
schema=QueryPlan,
)Discipline matters a great deal here, and it’s the difference between decomposition being a quality win versus a latency and cost liability. Cap the number of sub-queries, 3 is usually plenty for the compound questions that actually appear in real traffic, run them concurrently rather than sequentially so decomposition doesn’t multiply your latency budget, and only decompose at all when the structured call’s own classification says the question is genuinely compound. Decomposing “what’s the wifi password” into three unnecessary sub-queries is precisely how a well-intentioned latency budget quietly dies in production, one small overreach at a time.
A judgment call worth making explicit here, since it recurs later in the course: static decomposition, the pattern in this lesson, handles questions whose distinct parts are visible upfront, before any retrieval has happened. Questions whose second part genuinely depends on the first part’s answer, for instance “who manages the team that owns the billing service?”, where you need to find the service owner before you can even formulate the query about who manages them, are better served by the agentic loop covered in Module 6, which retrieves, reads what it found, and only then decides what to search for next. If you find yourself building increasingly elaborate multi-level decomposition trees to handle this kind of dependency chain, you’ve essentially rebuilt a worse, less flexible version of an agent, and you’d be better served adopting the real thing.
What happened to HyDE (a cautionary tale about techniques)
HyDE deserves its own section here because it teaches a broader lesson about how retrieval techniques age, one worth keeping in mind for every technique introduced elsewhere in this course too. In 2023, embedding models encoded queries and documents into the same vector space fairly naively, and a short five-word question would often embed quite far from the paragraph that actually answers it, simply because questions and answers are grammatically and structurally different kinds of text. HyDE’s fix was clever: ask an LLM to imagine what the answer document might look like, embed that imagined hypothetical document instead of the bare query, and search with that richer embedding. It worked, genuinely, and for a while it was a standard recommendation in every RAG tutorial.
Then the ground underneath it moved. Embedding models trained after 2024, the E5 and BGE lineage, Voyage, OpenAI’s v3 line, Qwen3-Embedding, use asymmetric training objectives and explicit instruction prefixes, literally prepending “query:” versus “passage:” during training, that close most of the gap HyDE was originally built to bridge. What remained of HyDE’s benefit no longer justified its costs: a full generative LLM call before every single search request, and a genuinely nasty failure mode where the model hallucinates a plausible-sounding but wrong hypothetical document, and retrieval then dutifully fetches evidence for that hallucination rather than for the user’s real question, actively making the wrong answer more confidently supported rather than less. By 2026 the technique survives mainly as a historical reference in older blog posts, not as production practice.
The durable lesson to carry forward, and it’s worth being explicit about this rather than leaving it implicit: query transformations that inject the user’s real, verifiable context, chat history, resolved entities, extracted filters, tend to age well as the underlying models improve around them. Transformations that inject the model’s own imagination in place of real context tend to age badly, because they’re essentially betting that the model’s guess is more useful than simply asking better of the retrieval system directly, a bet that gets worse as retrieval systems themselves improve. Keep that razor handy; the next lesson applies the exact same principle to the other half of query intelligence, pulling genuinely structured filters and routing decisions directly out of the query’s real content, so “PDFs from legal, updated this quarter” actually becomes a precise WHERE clause instead of a fuzzy semantic hope.