Design a RAG System: Interview Framework, Trade-offs, and Follow-ups

The direct answer is: design retrieval-augmented generation as four measurable stages - ingestion, retrieval, generation, and evaluation - with citations and a safe fallback when evidence is weak. Start with the product requirement, not a vector database or model name. Then explain how each stage can fail, how you will measure it, and which trade-offs the constraints force.

That structure gives an interviewer several useful signals. It shows that you understand RAG as a system rather than a prompt, that you can separate retrieval failures from generation failures, and that you can make choices without assuming one architecture fits every corpus.

If you need a broader foundation before this specialized case, read the ML system design interview guide. The framework below applies it specifically to grounded question answering.

1. Clarify the Product and Its Constraints

Suppose the prompt is: "Design an assistant that answers employee questions from internal documents." Do not begin by drawing embeddings. Ask what a correct product must do.

Clarify these dimensions:

• Users and tasks: Are users looking up policies, debugging technical incidents, or summarizing research? A precise policy lookup needs stronger citations than an exploratory summary.
• Corpus: What formats, languages, access controls, update frequency, and document volume are involved?
• Freshness: Must an edited document affect answers within minutes, or is a daily index acceptable?
• Latency and cost: Is this an interactive assistant or an offline report generator?
• Risk: Is an unsupported answer merely inconvenient, or could it lead to a security or compliance mistake?
• Success: What does a good answer mean to users? Correctness, evidence coverage, citation accuracy, latency, and abstention quality may all matter.

State an initial contract: "I will optimize for answers grounded in documents the user may access, include source citations, and abstain when retrieval does not provide enough evidence." That sentence defines the system more clearly than naming a model.

Turn the Contract Into Metrics

Use separate metrics for separate stages. For retrieval, evaluate whether relevant passages appear in the top results with recall at k or a judged relevance measure. For generation, evaluate factual support, answer completeness, citation correctness, and instruction adherence. For the product, track task completion, user corrections, latency, and cost per answered query.

Avoid claiming one automated score proves quality. Build a representative evaluation set with real question types, expected evidence, difficult negatives, stale documents, and questions that should be refused. Human review remains useful for nuanced answer quality.

2. Draw the End-to-End Architecture

A clear interview diagram has two paths.

Offline or asynchronous ingestion path:

1. Connect to approved document sources.
2. Parse text and structural metadata such as headings, tables, owners, timestamps, and permissions.
3. Normalize and split documents into retrievable units.
4. Create lexical indexes, embeddings, or both.
5. Store chunks, metadata, source links, versions, and access-control attributes.

Online query path:

1. Authenticate the user and interpret the query.
2. Apply permission and metadata filters.
3. Retrieve candidates.
4. Rerank and assemble context within the model's input budget.
5. Generate an answer constrained to the evidence.
6. Return citations and log quality signals without exposing sensitive content.

Keep the source of truth outside the index. The index is a derived serving layer that can be rebuilt. Use document IDs and versions so updates and deletions can propagate, and so every citation resolves to the exact source version used for the answer.

3. Design Ingestion Deliberately

Parsing and Chunking

Chunking is a retrieval decision, not cosmetic preprocessing. Very small chunks can match a query precisely but omit the context needed to answer it. Very large chunks preserve context but dilute the relevant signal and consume the generation budget.

Start with structure-aware chunks based on sections, paragraphs, or records. Preserve the document title and heading path as metadata. For a technical manual, a function description and its parameter table should probably stay together. For policy documents, keep a rule with its exceptions.

Overlap can prevent facts at boundaries from being lost, but it increases index size and may return duplicate evidence. Explain that you would tune chunk size and overlap against retrieval evaluation rather than choose them by habit.

Updates, Deletions, and Permissions

Assign a stable document ID and a version or content hash. On change, reprocess only affected documents, replace old chunks atomically, and invalidate caches tied to the previous version. Deletions must remove both text and vectors.

Permission handling is part of correctness. Attach access metadata during ingestion and enforce it before content reaches the generator. Filtering after generation is too late because unauthorized text may already have influenced the answer. If permissions are complex or frequently changing, retrieve candidate IDs and authorize them through a current policy service before fetching content.

4. Choose a Retrieval Strategy

Dense retrieval is useful when queries and passages use different words with similar meaning. Lexical retrieval is strong for exact identifiers, error codes, product names, and rare terms. Internal knowledge bases commonly contain both, so hybrid retrieval is a sensible default hypothesis, not a universal law.

A practical pipeline is:

1. Normalize or lightly rewrite the query when needed.
2. Run lexical and dense retrieval in parallel under metadata filters.
3. Merge candidate lists with a rank-fusion method.
4. Rerank a modest candidate set with a more accurate relevance model.
5. Deduplicate and select evidence that fits the context budget.

Explain the latency trade-off. A reranker can improve ordering but adds computation. Query rewriting can improve vague questions but can also alter intent. You can conditionally apply expensive steps only to ambiguous or low-confidence queries.

Concrete Interview Example

Consider the query: "Why does checkout fail with PX-104 after rotating keys?" Dense retrieval may understand the relationship to authentication, while lexical retrieval should preserve the exact error code. Metadata filters can limit results to the checkout service and current runbooks. The reranker should prefer a current incident guide over an old discussion that merely mentions the same code.

If the top evidence conflicts, do not silently blend it. Prefer authoritative and current sources where metadata supports that decision, or present the conflict with citations. A good answer names this failure mode before the interviewer asks.

5. Build Grounded Generation and Abstention

The prompt should define a narrow contract: answer from the supplied evidence, cite claims, distinguish inference from explicit facts, and say when the evidence is insufficient. Delimit retrieved content from system instructions because documents may contain text that looks like instructions. Treat retrieved text as untrusted data.

Context assembly should favor relevance, coverage, source authority, and diversity. Ten near-duplicate chunks from one document are often worse than a smaller set covering the main rule and its exceptions.

Abstention should use system evidence, not just the language model's self-reported confidence. Signals can include weak retrieval scores, disagreement among sources, missing required entities, or a verifier that cannot map claims to passages. The fallback might ask a clarifying question, return the most relevant documents, or state that no supported answer was found.

Do not promise hallucinations can be eliminated. The defensible goal is to reduce unsupported claims, detect risky cases, and make provenance visible.

6. Evaluate the System by Failure Layer

Create an evaluation set before optimizing. Include common questions, long-tail terminology, multi-document answers, permission boundaries, recent updates, ambiguous queries, adversarial document text, and unanswerable questions.

When an answer is wrong, classify the failure:

• Corpus failure: The needed source was never ingested or parsed correctly.
• Retrieval failure: The source exists but did not rank highly enough.
• Context failure: The right chunk was retrieved but dropped, truncated, or surrounded by distractors.
• Generation failure: The evidence was sufficient but the answer misused it.
• Policy failure: The system answered when it should have clarified or abstained.

This decomposition leads to targeted fixes. Fine-tuning the generator will not repair a missing document. Increasing top k will not repair poor parsing, and it may add distracting context.

For online monitoring, track ingestion lag, index freshness, empty-result rate, retrieval and generation latency, answer and citation feedback, abstention rate, token usage, and errors by query segment. Log source IDs and model or prompt versions for debugging, while applying retention and privacy controls.

Follow-up Questions to Expect

"Why not put the entire document in the prompt?"

For a tiny corpus, that may be the simplest baseline. At larger scale it raises latency and cost, exceeds context limits, and makes relevant evidence compete with noise. Retrieval also supports access filters and fresher updates. Say you would compare against the simple baseline rather than assume RAG is necessary.

"When would you fine-tune instead?"

Use retrieval primarily for changing or source-backed knowledge. Fine-tuning is better suited to consistent behavior, style, or task patterns when suitable training data exists. They can be combined. The deep learning and transformer interview guide covers related model-level questions.

"How do you reduce latency?"

Profile each stage first. Cache safe repeated computations, run independent retrieval paths concurrently, reduce candidate counts before reranking, use smaller models for query classification, stream generation, and skip query rewriting for clear requests. Preserve permission and freshness semantics when caching.

"How do you handle a new document?"

Parse, authorize, chunk, index, and publish it through a versioned pipeline. Define a freshness objective and monitor ingestion lag. Until publication succeeds, the source system remains authoritative and the assistant should not imply the new content is searchable.

A Strong Closing Summary

End the interview by connecting the choices: "I designed a permission-aware hybrid retrieval system because the corpus contains both semantic questions and exact identifiers. I would rerank candidates, generate only from selected evidence, cite source versions, and abstain on weak or conflicting support. I would launch with a simple evaluation set, diagnose errors by stage, and add complexity only where measured failures justify it."

That is a stronger conclusion than repeating component names. It shows priorities, trade-offs, and an iteration plan.

To practice presenting this architecture under follow-up pressure, use NeuraPrep's ML system design interview practice and get feedback on the reasoning in your answer at neuraprep.com.