Design a Recommendation System: ML Interview Answer Framework

To answer a recommendation system design question, move through six decisions in order: product objective, data and labels, candidate generation, ranking, serving, and evaluation with monitoring. Start with a simple baseline, make constraints explicit, and explain how feedback loops, cold start, and online experimentation affect the design.

This framework works for video, shopping, music, news, and social recommendations. The details change, but the reasoning path remains stable.

1. Clarify the Product Objective

Do not begin with collaborative filtering or a neural network. Ask what surface you are designing and what the product wants to improve.

Useful questions include:

• What is being recommended: videos, products, creators, or playlists?
• Is the surface a ranked feed, a related-items module, or notifications?
• What user action indicates value: a click, completed watch, purchase, save, or return visit?
• Which negative outcomes matter: quick abandonment, hides, returns, or reports?
• How fresh must recommendations be?
• What are the latency, traffic, catalog, and policy constraints?

Suppose the prompt is to design a home feed for a video application. Optimizing click-through alone can reward enticing thumbnails even when viewers leave immediately. A better objective may combine qualified watch time, completion, explicit satisfaction, and negative feedback, while applying safety and diversity constraints.

Name one primary product metric and a few guardrails. Then translate them into an ML task. The system might predict several user actions and combine them into a ranking score, rather than train on a single noisy label.

2. Define Training Data and Labels

Recommendation systems learn from exposure and response. A positive interaction means little without knowing which alternatives the user had an opportunity to choose.

For the video feed, useful events include:

• Impression with user, item, position, time, and request context.
• Click or playback start.
• Watch duration and completion.
• Like, save, share, hide, or report.
• Follow-up behavior such as continuing the session.

Join events using stable request and impression identifiers. Define attribution windows before creating labels. A purchase immediately after a product impression may be attributed differently from one several days later.

Avoid temporal leakage

Features and labels must represent what was available at recommendation time. Split training and validation data chronologically. Item popularity should be computed only from earlier events, not the complete day if the prediction occurred at its start. User history should end before the target impression.

Choose negatives carefully

An impressed but unclicked item is an observed negative for a click objective, though position and presentation affected the outcome. Random catalog items are easy negatives but may be so irrelevant that they teach little. Hard negatives retrieved for the user but not selected can sharpen ranking, while false negatives remain possible.

Account for biased feedback

The current recommender controls what gets exposed, so logs do not represent all user-item pairs. Position bias, popularity bias, and selection effects can reinforce the existing policy. Mention randomized exploration on a small, controlled portion of traffic or other debiasing approaches, with product and safety constraints.

3. Build Candidate Generation

Scoring every item for every request is usually impractical. Candidate generation reduces a large catalog to a manageable set with high recall. The ranker then spends more computation on those candidates.

Use several retrieval sources because each covers a different intent:

• Popular or trending items for broad appeal and fallback.
• Content-based retrieval using item metadata or embeddings.
• Collaborative retrieval from similar users or co-interactions.
• Subscribed creators, saved categories, or recently viewed themes.
• Fresh content and controlled exploration pools.

Merge results, remove unavailable or already consumed items where appropriate, enforce policy filters, and deduplicate before ranking.

Two-tower retrieval

A common learned approach encodes user context and items separately into the same vector space. Training makes positive pairs similar and sampled negatives less similar. Item vectors can be computed offline and indexed for approximate nearest-neighbor search; the user vector is computed at request time.

The separation makes retrieval fast but limits fine-grained interactions between each user and item. That is acceptable at the high-recall candidate stage. A later ranker can model richer feature crosses.

Candidate-generation evaluation

Measure whether the item a user engaged with appears among retrieved candidates using recall at k, segmented by user and item cohorts. Also inspect source contribution, duplication, coverage, freshness, and retrieval latency. If relevant items never reach the ranker, improving the ranker cannot recover them.

4. Design the Ranker

The ranker scores hundreds or thousands of candidates using user, item, and context features. Start with a baseline such as a weighted heuristic or gradient-boosted model. Add complexity only when evaluation demonstrates a meaningful gap.

Feature groups for the video example include:

• User: long-term category preferences, creator affinity, activity level, and historical session patterns.
• Item: topic, language, age, duration, quality signals, and aggregate engagement computed without leakage.
• Context: time, device, network, current session actions, and entry surface.
• Cross features: user-topic affinity, recent creator frequency, and similarity to the current session.

Multi-objective prediction

The model could predict click, expected watch, completion, like, and negative feedback separately. A policy layer combines calibrated outputs into a score. This keeps tradeoffs visible and allows product changes without relabeling one opaque target.

Be careful when combining probabilities and continuous outcomes with different scales. Describe calibration and offline simulation, but emphasize that online experiments determine product impact.

Reranking and constraints

The highest individual scores may produce a repetitive slate. A reranker can account for the list as a whole by limiting repeated creators, improving topic diversity, reserving fresh inventory, and enforcing policy or availability constraints. Explain which are hard constraints and which are scored tradeoffs.

Ranking metrics such as NDCG or mean reciprocal rank capture order better than classification accuracy, but the choice should match the surface. The broader ML system design interview guide explains how to connect offline metrics with business goals.

5. Design Online Serving

Draw the request path clearly:

1. The client requests recommendations with authenticated user and context.
2. The service fetches online state and cached user features.
3. Candidate sources run in parallel within bounded timeouts.
4. Results are merged, filtered, and deduplicated.
5. The ranking service fetches candidate features and scores the set.
6. A reranker applies slate constraints.
7. The service logs recommendations with request and model versions.
8. The client logs impressions only when items are actually shown.

Offline and online features

Compute stable features such as long-term user preferences and item aggregates in batch or streaming pipelines, then store them for low-latency access. Compute current-session signals online. The training pipeline and serving pipeline must share definitions to avoid training-serving skew.

Point-in-time correctness matters for training, while freshness and fallback behavior matter online. If a feature is unavailable, use a defined default and emit a metric rather than silently changing input shape.

Latency and graceful degradation

Assign a budget to retrieval, feature fetch, ranking, and reranking. If one personalized candidate source times out, return results from other sources or a cached fallback. A recommendation surface can often degrade to popular eligible items, but it should not bypass policy filters.

Cache item features and reusable candidate sets where safe. Avoid caching a final personalized response without considering user state, inventory changes, and access rules.

Model deployment

Version model artifacts, feature schemas, and score-combination policy. Validate compatibility before serving. Use shadow evaluation or a small traffic allocation for a new model, monitor technical guardrails, and retain a fast rollback path. Log enough version information to reproduce a decision.

6. Evaluate, Experiment, and Monitor

Offline evaluation

Use a chronological holdout and compare against the current system and a simple popularity baseline. Depending on the task, track recall at k for retrieval, NDCG at k or ranking loss for ordering, calibration for predicted actions, catalog coverage, and cohort slices.

Offline replay cannot fully measure a new policy because logged data came from the old policy. State that limitation rather than overclaiming from held-out metrics.

Online experimentation

Run a controlled experiment with stable assignment, usually at the user level for a personalized experience. Track the primary metric plus guardrails such as negative feedback, latency, errors, creator or catalog coverage, and longer-term behavior when relevant. Define exposure, sample, duration, and stopping rules with the experimentation team before launch.

Do not assume a statistically significant movement is automatically worthwhile. Consider effect size, operational cost, and whether one cohort gained at another's expense.

Production monitoring

Monitor the complete system, not only model scores:

• Event and feature freshness, missingness, and schema changes.
• Candidate volume, source contribution, and retrieval latency.
• Score and prediction distributions by model version.
• Serving latency, timeout rate, errors, and fallback use.
• Engagement and negative outcomes by important cohorts.
• Catalog concentration, freshness, and repeated-content rates.

Set alerts around actionable failure modes. A sudden drop in candidates from one source may indicate an index or permission issue even when the endpoint still returns successful responses.

Worked Interview Example: Video Home Feed

Here is a concise answer you can expand under follow-up questions.

Goal: Rank an eligible set of videos for each home-feed request to improve qualified watch and satisfaction, with guardrails for hides, reports, latency, and content diversity.

Data: Log shown impressions, position, clicks, watch outcomes, and explicit feedback. Build labels with defined windows and chronological splits. Use only features available before each impression.

Retrieval: Combine subscriptions, two-tower embedding search, topic affinity, popular eligible videos, and a small fresh-content pool. Filter policy violations and deduplicate.

Ranking: Begin with a gradient-boosted baseline, then test a multi-task model predicting click, watch, completion, and negative feedback. Calibrate outputs and combine them through an explicit product policy. Rerank for creator and topic diversity.

Serving: Fetch session state, run candidate sources in parallel, batch feature access, score, rerank, and return within the agreed latency budget. Fall back to cached eligible popularity results if personalization dependencies fail.

Evaluation: Measure retrieval recall, ranking quality, calibration, coverage, and cohort behavior offline. Validate product impact with user-level experiments. Monitor data freshness, source health, latency, score drift, concentration, and negative feedback.

That answer covers the end-to-end path while leaving room for the interviewer to select a deeper topic.

Follow-Up Questions to Prepare

How do you handle a new user?

Ask for lightweight preferences when appropriate, then use context, regional or broad popularity, and exploration. Adapt quickly from session actions. Avoid pretending a detailed user embedding exists before evidence does.

How do you handle a new item?

Use content metadata and embeddings, quality or policy checks, and a controlled exploration allocation. Track whether the item receives enough eligible exposure to collect feedback.

How do you prevent filter bubbles?

Define the product concern precisely. Add diverse candidate sources, slate-level diversity, and controlled exploration, then measure concentration and satisfaction. Diversity is not one universal metric, so tie it to item, creator, or topic exposure as appropriate.

What if popularity dominates the model?

Inspect labels, negative sampling, exposure bias, and item-frequency features. Compare cohort and catalog coverage. Possible responses include debiased training, regularization, capped popularity features, exploration, or reranking constraints, each with a relevance tradeoff.

Would you use an LLM in the recommender?

Only where it solves a defined problem, such as deriving content representations or interpreting sparse text. Consider inference cost, latency, safety, and evaluation. The central retrieval-ranking architecture does not require an LLM.

Common Interview Mistakes

• Choosing a model before defining the user objective.
• Treating unexposed items as definite negatives.
• Discussing ranking while skipping candidate generation.
• Using random train-test splits for temporal interactions.
• Ignoring cold start, feedback loops, or policy filters.
• Naming infrastructure components without explaining their purpose.
• Reporting offline accuracy for an ordered recommendation surface.
• Saying the system will be monitored without naming signals and responses.

Practice the framework aloud with different surfaces. Change the objective, catalog size, freshness requirement, and cost of a bad recommendation, then adapt the design. Use NeuraPrep to rehearse ML system design questions and receive feedback on the structure and clarity of your answer.