LLM Engineer Interview Questions: 25 Questions with Answer Guides

An LLM engineer interview tests whether you can turn a capable but unreliable model into a useful production system. Prepare to explain transformer fundamentals, choose between prompting, retrieval, and fine-tuning, define evaluation criteria, and reason about latency, cost, safety, and failure recovery.

The strongest answers follow a simple pattern: state the mechanism, connect it to the product constraint, name the tradeoff, and describe how you would verify the decision. The 25 questions below are designed for that kind of answer, not rote definitions.

Transformer and LLM Fundamentals

1. How does scaled dot-product attention work?

Answer guide: Each token representation is projected into a query, key, and value. Query-key dot products measure compatibility, division by the square root of the key dimension keeps logits from becoming too large, and softmax converts them to weights over values. Mention masking for decoder-only generation and the quadratic cost in sequence length. A complete answer explains why scaling matters, not only the formula.

2. Why does an autoregressive LLM use a causal mask?

Answer guide: The mask prevents a position from attending to future tokens during training. That makes parallel teacher-forced training consistent with left-to-right inference, where future tokens are unavailable. Without it, the model could leak the answer from later positions and minimize training loss without learning the generation task.

3. What is the purpose of positional information?

Answer guide: Attention alone does not encode token order. Positional representations make sequence order and distance available to the model. Contrast absolute embeddings with relative methods such as rotary position embeddings. Discuss extrapolation carefully: supporting a larger input tensor does not guarantee quality beyond lengths represented during training.

4. What changes when temperature is increased during decoding?

Answer guide: Temperature divides logits before softmax. A lower value sharpens the distribution and makes outputs more deterministic; a higher value flattens it and increases diversity and error risk. Temperature is not a confidence threshold. For factual extraction, use low randomness and constrained output; for ideation, some diversity may be useful.

5. Compare greedy decoding, beam search, top-k, and top-p sampling.

Answer guide: Greedy decoding selects the highest-probability token each step. Beam search retains several high-scoring sequences and is useful where sequence likelihood is meaningful, but can produce repetitive text. Top-k samples from a fixed number of candidates. Top-p samples from the smallest set whose cumulative probability reaches a threshold, adapting to distribution shape. Choose based on determinism, diversity, and latency needs.

6. Why can an LLM hallucinate?

Answer guide: Next-token training rewards plausible continuation, not factual verification. Hallucination can arise from missing knowledge, ambiguous context, retrieval failure, misleading instructions, or decoding randomness. Mitigations should match the cause: better context and citations for knowledge gaps, tool calls for live facts, abstention rules for uncertainty, and evaluation against representative requests. Do not claim hallucination can be eliminated.

For deeper architecture review, use the deep learning and transformer interview questions alongside these production-focused questions.

Prompting, Tools, and Structured Output

7. How would you design a reliable prompt?

Answer guide: Separate trusted instructions from untrusted content, define the task and audience, specify output constraints, and add only examples that clarify ambiguous behavior. Keep stable prompt components versioned. Test the prompt against normal, adversarial, and edge-case inputs rather than polishing one demonstration until it works.

8. When does few-shot prompting help?

Answer guide: It helps when the task, label semantics, tone, or output shape is difficult to express as rules. Examples should represent important variation and common boundaries. They also consume context and can bias outputs toward superficial patterns, so compare against a zero-shot baseline and evaluate whether each example earns its cost.

9. How do you make structured output dependable?

Answer guide: Prefer schema-constrained generation or tool calling when the model API supports it. Validate types and allowed values after generation, reject unknown fields, and retry only errors that can plausibly be repaired. Never treat syntactically valid JSON as semantically valid. For a support-ticket classifier, verify that the category belongs to the current taxonomy before writing it downstream.

10. What is prompt injection, and how do you defend against it?

Answer guide: Prompt injection is untrusted text attempting to override instructions or trigger unauthorized actions. Treat retrieved documents and user content as data, not authority. Enforce permissions outside the model, expose narrow tools, validate arguments, require confirmation for consequential actions, and avoid placing secrets in model context. Prompt wording alone is not a security boundary.

11. How would you design an LLM agent with tools?

Answer guide: Start with a bounded workflow before an open-ended loop. Define each tool with a narrow schema, explicit authorization, timeouts, idempotency where possible, and observable results. Limit steps and cost. For a calendar assistant, reading availability and creating an event should be separate tools, with confirmation before the write.

Retrieval-Augmented Generation

12. When should you use RAG instead of fine-tuning?

Answer guide: Use retrieval when answers depend on changing, private, or source-grounded knowledge. Fine-tuning is better suited to behavior, style, or repeated task patterns. They can be combined, but neither fixes a weak problem definition. If an internal policy changes weekly and answers need citations, retrieval is the natural first choice.

13. How would you build a basic RAG pipeline?

Answer guide: Ingest and normalize documents, split them into coherent chunks, attach access-control and source metadata, create embeddings, and index them. At query time, retrieve candidates, optionally rerank them, assemble bounded context, generate an answer, and preserve citations. Log each stage so a bad answer can be traced to retrieval, context construction, or generation.

14. How do you choose a chunking strategy?

Answer guide: Chunk by semantic structure when possible: sections, paragraphs, records, or code units. Small chunks improve retrieval specificity but may lose context; large chunks preserve context but add noise and token cost. Test chunk size and overlap using real questions, including questions whose evidence crosses a section boundary.

15. What is hybrid retrieval?

Answer guide: Hybrid retrieval combines lexical matching with dense semantic retrieval. Lexical search is strong for exact identifiers, product names, and rare terms; embeddings help with paraphrases and conceptual similarity. Merge or rerank candidates, then measure retrieval recall on a labeled query set rather than assuming one approach always wins.

16. How do you evaluate retrieval separately from generation?

Answer guide: Build queries with known relevant documents or passages. Measure whether the evidence appears in the top results using recall at k, rank-sensitive metrics, and manual error categories. Then evaluate answer faithfulness and usefulness with the retrieved context held fixed. Separating stages tells you whether to improve the retriever or the generator.

17. How would you handle document permissions in RAG?

Answer guide: Enforce access control during retrieval, using the authenticated user's identity and document metadata. Do not retrieve everything and ask the LLM to hide forbidden passages. Carry tenant and permission filters through caches, indexes, and logs, and test for cross-tenant leakage. This is an application security requirement, not a prompt instruction.

Fine-Tuning and Model Adaptation

18. Compare full fine-tuning with parameter-efficient fine-tuning.

Answer guide: Full fine-tuning updates all weights and offers maximum flexibility at high compute, storage, and operational cost. Parameter-efficient methods update small adapters or low-rank matrices while freezing the base model, making experiments and per-task variants cheaper. Discuss data quality and evaluation first; an efficient method trained on inconsistent labels remains a poor model.

19. What makes a good instruction-tuning dataset?

Answer guide: Examples should reflect real task distribution, have consistent instructions and responses, include difficult boundaries, and exclude sensitive or duplicated data. Split by source or time when near-duplicates could leak across train and test. Review label guidelines and disagreements. A smaller coherent set can be more useful than a larger noisy set.

20. What is preference optimization trying to achieve?

Answer guide: Preference methods use comparisons between responses to steer behavior toward desired qualities that next-token likelihood does not capture directly. Explain the source and limits of preference labels, including annotator inconsistency and reward misspecification. Evaluation must check the desired behavior and regressions such as verbosity, refusal errors, or loss of factuality.

21. How would you decide whether fine-tuning succeeded?

Answer guide: Define a frozen test set and task-specific rubric before training. Compare the tuned model with the prompted baseline on quality, safety, latency, and cost. Slice results by request type and inspect regressions. A lower training loss is not a product result, and model-based grading should be calibrated with human review.

Evaluation, Serving, and Production

22. How do you evaluate an open-ended LLM application?

Answer guide: Combine deterministic checks, reference-based metrics where appropriate, rubric-based human review, and calibrated model graders. Build a representative set from actual use cases and adversarial boundaries. Track dimensions separately, such as correctness, groundedness, instruction following, tone, and refusal quality, because one aggregate score hides tradeoffs.

23. How would you reduce LLM application latency?

Answer guide: Measure time to first token and generation time separately. Reduce unnecessary context, retrieve fewer but better passages, choose an appropriately sized model, stream responses, cache safe repeated work, parallelize independent retrieval or tool calls, and cap output length. Explain the quality cost of each optimization rather than promising free speed.

24. What would you monitor in production?

Answer guide: Monitor request volume, errors, latency, token use, cost, tool failures, retrieval quality proxies, safety events, and user outcomes. Sample traces with privacy controls so prompts, retrieved context, tool calls, and outputs can be debugged. Watch distributions by model and prompt version. User thumbs-up alone is too sparse and biased to serve as the only quality signal.

25. Design a support assistant over a private knowledge base.

Answer guide: Clarify users, data sensitivity, freshness, supported actions, and success criteria. Propose permission-aware hybrid retrieval, reranking, cited answers, and an explicit fallback when evidence is insufficient. Keep ticket updates behind validated tools and user confirmation. Evaluate retrieval recall, grounded answer quality, escalation accuracy, latency, and cost. Roll out to a small scope, inspect failure traces, and expand only after the guardrails work.

This final question is a compact system design exercise. Use the broader ML system design interview framework to structure requirements, data, modeling, serving, and monitoring.

How to Practice These Questions

Do not memorize the answer guides word for word. Pick five questions from different sections and answer each in two minutes. Then add one follow-up: what changes under strict latency, private data, weak labels, or a smaller budget? For design questions, sketch the request path and identify where failures can be observed.

A strong preparation loop is simple:

1. Give a direct answer in the first sentence.
2. Explain the mechanism with one concrete example.
3. State a tradeoff or failure mode.
4. Define how you would evaluate the choice.
5. Revisit weak answers after a day without notes.

When you are ready to practice under interview conditions, use NeuraPrep for focused AI and ML interview questions and feedback on how clearly you reason through the answer.