Deep Learning & Transformer Interview Questions: Technical Q&A with Worked Answers

If you've made it to the technical ML interview, you already cleared the resume screen. What happens next depends on whether you understand the material deeply or just recognize the terminology. Interviewers can tell the difference within two questions.

This guide covers deep learning and transformer interview questions as they actually appear in senior ML and AI Engineer interviews — with full technical answers, not hand-wavy summaries. Work through these as practice, not as reading material.

Neural Network Foundations

Why do deep networks suffer from vanishing gradients, and how do you fix it?

The vanishing gradient problem occurs during backpropagation when gradients are repeatedly multiplied by small values as they flow through layers. If the activation function's derivative is consistently less than 1 — which is true for sigmoid over most of its range — the gradient signal shrinks exponentially with depth.

For sigmoid: the derivative is $\sigma'(x) = \sigma(x)(1 - \sigma(x))$, which has a maximum value of 0.25. Stack 20 layers and you're multiplying 20 terms each at most 0.25 — the gradient at layer 1 is effectively zero.

Fixes:

• ReLU activations: derivative is either 0 or 1. No shrinkage for positive activations. Note that ReLU has its own problem (dying ReLU: neurons that enter the negative regime stay dead), which Leaky ReLU and ELU address.
• Residual connections: shortcut connections (as in ResNet) allow gradients to flow directly through addition, bypassing the multiplicative chain.
• Batch normalization: normalizes activations to prevent them from saturating into the flat regions of activation functions.
• Careful initialization: Xavier or He initialization sets initial weights such that variance is preserved through forward and backward passes.

What does batch normalization actually do, and when would you not use it?

Batch normalization normalizes the activations of a layer across the batch dimension, then applies learned scale ($\gamma$) and shift ($\beta$) parameters:

$$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad y_i = \gamma \hat{x}_i + \beta$$

This keeps activations in a stable range during training, reduces sensitivity to initialization, and provides mild regularization (because the normalization introduces noise from the batch statistics).

When not to use it:

• Small batch sizes: with batches of 4–8, the batch statistics are too noisy. Use layer normalization or group normalization instead.
• Recurrent networks: batch norm interacts poorly with variable-length sequences and the temporal structure of RNNs. Layer normalization is standard here.
• Transformers: layer normalization is used, not batch normalization. The input sequence length varies, and normalizing across the batch doesn't make sense semantically.

What is dropout and why does it work?

Dropout randomly zeros out activations with probability $p$ during training. At test time, all neurons are active but weights are scaled by $(1-p)$ to maintain expected values.

The standard explanation is that dropout prevents co-adaptation — neurons cannot rely on specific other neurons always being present, so they learn more robust, independent features.

A more rigorous framing: dropout is approximately training an ensemble of $2^n$ thinned networks (where $n$ is the number of units) and averaging their predictions at test time. This ensemble effect explains the regularization benefit.

In practice: dropout rates of 0.2–0.5 work well for fully connected layers. It's rarely used in convolutional layers (spatial dropout is preferred there) and not used in batch normalization layers.

CNN & Sequence Modeling Questions

What does a convolutional layer actually compute, and why does it generalize better than a fully connected layer for images?

A convolution applies a learned filter across all spatial positions of the input, computing a dot product at each position:

$$(f * x)[i, j] = \sum_{k,l} f[k, l] \cdot x[i+k, j+l]$$

Two properties matter for images:

Translation equivariance: if the input shifts, the output shifts by the same amount. A cat detector doesn't need to learn a separate detector for every possible position — the same filter handles it everywhere.

Parameter sharing: the same filter weights apply at every spatial position. A 3x3 filter on a 224x224 image uses 9 parameters, not $224^2 \times 9$. This is why CNNs generalize from limited data where fully connected layers would overfit.

Why did LSTMs replace vanilla RNNs, and why are they now mostly replaced by transformers?

Vanilla RNNs update hidden state as $h_t = \tanh(W_h h_{t-1} + W_x x_t)$. The problem: gradients through this recurrence vanish (or occasionally explode) over long sequences, because the same weight matrix $W_h$ is applied at every step.

LSTMs address this with a cell state $c_t$ that acts as a protected memory, updated through gating mechanisms (input, forget, output gates). The forget gate $f_t = \sigma(W_f [h_{t-1}, x_t])$ controls what to erase; the gradient can flow through $c_t$ without being repeatedly multiplied by weights, which alleviates vanishing gradients over moderate sequence lengths.

Why transformers displaced them: LSTMs still process sequences sequentially — step $t$ depends on step $t-1$. This prevents parallelization during training. Transformers compute attention over all positions simultaneously, which scales better with modern hardware and handles very long-range dependencies more directly.

Transformer Architecture: The Hard Questions

Derive how self-attention is computed.

Self-attention maps each input token to three vectors: Query ($Q$), Key ($K$), and Value ($V$), all computed by learned linear projections of the input embeddings:

$$Q = XW^Q, \quad K = XW^K, \quad V = XW^V$$

The attention score between token $i$ and token $j$ is the dot product of their query and key vectors, scaled and passed through softmax:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

The $\sqrt{d_k}$ scaling prevents the dot products from growing large in magnitude for high-dimensional keys, which would push the softmax into regions with very small gradients.

The output for each token is a weighted sum of all value vectors, where the weights come from how much attention that token pays to every other token. Every token attends to all other tokens simultaneously — this is why transformers have no sequential dependency and can be parallelized.

Why do transformers need positional encoding?

Self-attention is permutation-invariant. Shuffle the input tokens and the attention scores change, but the operation itself has no notion of which token came first. Given the sentence "the dog bit the man" vs. "the man bit the dog," without positional information the model would compute identical representations.

Positional encodings inject position information into the embeddings before attention is computed:

$$\text{PE}(pos, 2i) = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$\text{PE}(pos, 2i+1) = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

Using sinusoidal functions at different frequencies means the model can represent absolute position, and the dot product of two positional encodings captures relative distance. Modern models (RoPE, ALiBi) use learned or relative positional encodings that generalize better to sequence lengths not seen during training.

What happens if you remove positional encoding entirely? The model becomes a bag-of-words at the sequence level. Syntax and word order disappear. For tasks where order matters — translation, generation, most sequence classification — performance collapses.

What is the purpose of multi-head attention?

Single-head attention produces one set of attention weights per token pair. Multi-head attention runs $h$ attention operations in parallel, each with its own $W^Q$, $W^K$, $W^V$ projections:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W^O$$

The motivation is that different heads can attend to different types of relationships simultaneously. In practice, learned heads specialize: some attend to syntactic relationships (subject-verb), others to semantic proximity, others to coreference. Single-head attention must trade off between these.

The dimensions are maintained by projecting each head into $d_k = d_{model}/h$, so the total parameter count and computation is roughly the same as single-head attention at full dimension.

What is the architectural difference between BERT and GPT?

Both use transformer layers. The difference is in training objective and architecture:

GPT uses a decoder-only transformer with causal (left-to-right) masking. Each token can only attend to previous tokens. Trained via next-token prediction. Naturally suited for generation tasks.

BERT uses an encoder-only transformer with bidirectional attention — each token attends to all other tokens. Trained via masked language modeling (predict randomly masked tokens) and next sentence prediction. Suited for understanding tasks (classification, named entity recognition, question answering) where you have the full context available at inference time.

Practical implication: if you're classifying text, fine-tune BERT-style models. If you're generating text, use GPT-style models. For tasks that require both (summarization, translation), encoder-decoder architectures (T5, BART) are the natural fit.

LLM & Fine-Tuning Questions

What is the difference between pre-training and fine-tuning?

Pre-training is training a model on a massive corpus with a self-supervised objective (next-token prediction, masked language modeling). No task-specific labels required. This is expensive — GPT-4 training reportedly cost >$100M in compute.

Fine-tuning takes a pre-trained model and adapts it to a specific task or domain using a smaller labeled dataset. The model's weights are updated, but starting from the pre-trained initialization means it converges faster and with less data than training from scratch.

Variants:

• Full fine-tuning: all weights updated. Most flexible, most expensive.
• LoRA (Low-Rank Adaptation): freeze base model weights, add trainable low-rank matrices to attention layers. Reduces trainable parameters by ~100x with minimal performance loss. Standard for LLM fine-tuning.
• Adapter layers: insert small trainable modules between frozen layers.

What is RLHF and why is it used?

Reinforcement Learning from Human Feedback trains a reward model on human preference comparisons (response A vs. response B, which is better?), then uses that reward model to fine-tune the LLM via reinforcement learning (typically PPO).

The motivation: pure next-token prediction on internet text teaches models to generate likely text, not helpful or safe text. Internet text contains misinformation, toxicity, and unhelpful patterns. RLHF steers the model toward outputs humans prefer.

The limitation: the reward model is imperfect, and the LLM can overfit to it — a phenomenon called reward hacking. The model finds outputs that score well on the reward model but are not genuinely better (verbose responses, sycophantic tone). Constitutional AI and Direct Preference Optimization (DPO) are alternatives that reduce some of these issues.

When should you prompt-engineer vs. fine-tune?

Prompt engineering first. It's fast, cheap, and reversible. If you can achieve acceptable performance by carefully structuring your prompt, adding few-shot examples, or using chain-of-thought reasoning, do that.

Fine-tune when:

• The task requires knowledge or style not achievable through prompting alone
• You need consistent output format that prompting can't reliably enforce
• Latency or cost requires a smaller model that a larger prompted model is substituting for
• You have > a few hundred labeled examples and performance still falls short

One misconception: fine-tuning doesn't add new knowledge as effectively as pre-training. It shapes behavior and adapts style. For injecting large bodies of factual knowledge, RAG (retrieval-augmented generation) is typically more reliable.

Practical Interview Tips

On math questions: Interviewers asking you to derive the attention formula are not testing memory. They want to see if you understand why the scaling factor $\sqrt{d_k}$ is there, or what goes wrong without it. Derive it structurally, explain the design decisions.

On architecture questions: The BERT vs. GPT question is a favorite not because the answer is obscure, but because candidates who only read about transformers without building intuition around them will give a vague answer. Know the masking difference and what it implies for use cases.

On LLM questions: This area is evolving fast and interviewers know it. You're not expected to know every paper. You are expected to reason about trade-offs clearly — fine-tuning vs. RAG, RLHF vs. DPO — without pretending certainty where there is none.

The best candidates don't recite answers. They reason out loud, flag their uncertainty, and engage with follow-up questions rather than retreating to memorized scripts.

Practice that reasoning skill specifically. NeuraPrep's AI-powered interview simulations for deep learning and transformer questions put you in the hot seat with follow-up questions — the format that actually builds interview readiness. Try it at neuraprep.com.