Machine Learning Evaluation Metrics Interview Guide: Precision, Recall, F1, ROC-AUC, and PR-AUC
The short answer is: there is no universally best machine learning evaluation metric. Choose a metric from the decision the model supports, the costs of false positives and false negatives, class prevalence, whether you need ranking or calibrated probabilities, and the operating threshold. In an interview, defining a metric is only the start; a strong answer explains why it matches the product decision and where it can fail.
This guide focuses on classification metrics commonly tested in data scientist, machine learning engineer, and AI engineer interviews. It includes a worked example, threshold selection, calibration, and concise answers to common follow-ups.
Which Classification Metric Should You Use?
| Metric | What it measures | Use it when | Main limitation |
|---|---|---|---|
| Accuracy | Fraction of all predictions that are correct | Classes are reasonably balanced and error costs are similar | Can hide failure on a rare class |
| Balanced accuracy | Average recall across classes | You want equal class-level importance under imbalance | Does not encode business costs |
| Precision | Fraction of predicted positives that are correct | False positives are costly | Ignores false negatives |
| Recall | Fraction of actual positives detected | False negatives are costly | Ignores false positives |
| F1 | Harmonic mean of precision and recall | You need one threshold-specific score and value precision and recall similarly | Ignores true negatives and unequal costs |
| ROC-AUC | Ranking quality across thresholds using TPR versus FPR | You want a threshold-independent ranking comparison | Can look optimistic with rare positives |
| PR-AUC | Precision-recall trade-off across thresholds | The positive class is rare and its retrieval quality matters | Depends on class prevalence |
| Log loss | Quality of predicted probabilities, heavily penalizing confident errors | Probabilities themselves matter | Sensitive to badly calibrated extreme predictions |
| Brier score | Mean squared error of predicted probabilities | You care about probability accuracy and calibration | Mixes calibration and discrimination effects |
The interview-safe rule is: begin with the action and error costs, not the metric name. A fraud model that sends cases to a limited review queue has a different objective from one that automatically blocks transactions, even if both predict fraud.
Start With the Confusion Matrix
For binary classification, define the positive class first. Then every thresholded prediction has one of four outcomes:
- True positive (TP): predicted positive and actually positive.
- False positive (FP): predicted positive but actually negative.
- False negative (FN): predicted negative but actually positive.
- True negative (TN): predicted negative and actually negative.
The positive class is a modeling convention, not a synonym for a desirable outcome. In fraud detection, fraud is usually positive. In a quality-control model, a defective item may be positive. State the convention before calculating anything.
The core formulas are:
$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$
$$\text{Precision} = \frac{TP}{TP + FP}$$
$$\text{Recall} = \frac{TP}{TP + FN}$$
$$\text{Specificity} = \frac{TN}{TN + FP}$$
Google's classification metrics guide uses the same definitions and emphasizes that the useful metric depends on the task, error costs, and class balance.
Worked Example: Calculate Accuracy, Precision, Recall, and F1
Suppose a model evaluates 1,000 transactions and produces:
- TP = 40 fraudulent transactions caught
- FP = 20 legitimate transactions flagged
- FN = 10 fraudulent transactions missed
- TN = 930 legitimate transactions passed
Then:
- Accuracy = (40 + 930) / 1,000 = 97%
- Precision = 40 / (40 + 20) = 66.7%
- Recall = 40 / (40 + 10) = 80%
- F1 = 2 x (0.667 x 0.8) / (0.667 + 0.8) = 72.7%
The interpretation matters more than the arithmetic. Of all flagged transactions, about two thirds were actually fraudulent. Of all fraudulent transactions, the model caught four fifths. Whether this is acceptable depends on the cost of missed fraud, the harm of interrupting legitimate customers, and review capacity.
Precision vs Recall: What Is the Difference?
Precision asks: when the model predicts positive, how often is it right? Recall asks: of all real positives, how many did the model find?
Prioritize precision when false positives are especially costly. Examples include automatically removing legitimate content, blocking a legitimate payment, or sending an expensive intervention.
Prioritize recall when false negatives are especially costly. Examples include an initial safety screen, detecting a dangerous defect, or retrieving candidates for a later ranking stage.
Precision and recall commonly trade off as the classification threshold changes. Raising the threshold usually predicts fewer positives: false positives often fall, but false negatives often rise. Lowering it usually does the reverse. This is why saying "maximize both" is incomplete unless the model can improve its ranking or representation, not merely move its threshold.
When Is Accuracy Misleading?
Accuracy can be misleading when classes are imbalanced or mistakes have unequal costs. If only 1% of examples are positive, a model that always predicts negative achieves 99% accuracy while detecting no positives.
That does not make accuracy inherently bad. It can be useful when classes are balanced, labels are reliable, and error costs are similar. A strong interview answer avoids absolutes: inspect the confusion matrix, compare against a simple baseline, and report metrics aligned with the actual decision.
Balanced accuracy is one alternative under imbalance. It averages recall across classes, preventing a large majority class from dominating the score. But it still assigns equal importance to classes rather than expressing monetary, safety, or capacity costs.
What Is the F1 Score, and When Should You Use It?
F1 is the harmonic mean of precision and recall:
$$F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$
Because the harmonic mean is pulled toward the lower input, F1 is high only when both precision and recall are high. It is useful as a compact threshold-specific comparison when the positive class matters and precision and recall have roughly similar importance.
F1 is not a business objective by default. It ignores true negatives, treats precision and recall symmetrically, and does not represent different error costs. If recall matters more, an F-beta score with beta greater than one weights recall more heavily. If precision matters more, beta below one weights precision more heavily. Better still, select a threshold using explicit costs or operational constraints when those are available.
ROC-AUC vs PR-AUC
What Does ROC-AUC Measure?
The ROC curve plots true positive rate, which is recall, against false positive rate across thresholds. ROC-AUC summarizes that curve. It can also be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative.
ROC-AUC evaluates ranking, not the quality of predicted probabilities and not performance at one deployed threshold. A model can have good ROC-AUC yet be poorly calibrated or unsuitable at the operating region the product needs.
What Does PR-AUC Measure?
The precision-recall curve plots precision against recall as the threshold changes. It focuses on performance for the positive class and is often more informative when positives are rare.
PR performance depends on prevalence: a random classifier's expected precision follows the positive-class rate. This makes PR curves sensitive to the evaluation population. Compare models on the same representative test distribution and report the baseline prevalence.
Also clarify implementation terminology. Average precision and the trapezoidal area under a precision-recall curve are related but can produce different values. State which calculation a library or report uses; scikit-learn documents both in its model evaluation guide.
Interview Answer: Which One Should You Choose?
Use ROC-AUC when overall ranking across positives and negatives is the relevant comparison and both classes receive meaningful attention. Prefer a precision-recall view when positives are rare and you care about retrieving them without overwhelming the system with false positives. In either case, also evaluate performance at realistic thresholds and by important data segments.
How Should You Choose a Classification Threshold?
A classification model often outputs a score or probability. The threshold converts that value into an action. A default threshold of 0.5 is not automatically optimal.
Use this interview framework:
- Define the action triggered by a positive prediction.
- Estimate or rank the costs of FP and FN outcomes.
- Add operational constraints such as review capacity, latency, or intervention budget.
- Inspect candidate thresholds on validation data using the resulting utility or constraints.
- Confirm the selected threshold once on untouched test data.
- Monitor prevalence, score distributions, calibration, outcomes, and segment performance after launch.
For a review queue that can process 1,000 cases daily, useful criteria might be precision among the top 1,000 scores and recall captured within that capacity. For an automated block, the acceptable false-positive rate may be far lower. The model can be identical while the threshold and evaluation change with the action.
Do not tune a threshold on the test set and then report that same test result as an unbiased estimate. Threshold selection is part of model development. Google's thresholding guide illustrates how changing a threshold changes every confusion-matrix count.
Ranking Is Not Calibration
A model is well ranked if positives tend to receive higher scores than negatives. It is calibrated if predictions near 0.8 are positive approximately 80% of the time in the relevant population. These are different properties.
ROC-AUC can remain unchanged under a monotonic transformation of scores, while calibration changes. If probabilities drive pricing, expected loss, staffing, or risk, evaluate them with reliability diagrams and proper scoring rules such as log loss or Brier score.
Calibration can also vary by segment and over time. Report overall calibration alongside decision-relevant groups, but ensure each estimate has enough data to be meaningful.
Three Common Interview Scenarios
1. Safety Detection
If missing a positive is much worse than reviewing a false alarm, prioritize high recall subject to a tolerable false-positive burden. Report the actual operating threshold, not only an area-under-curve score.
2. Automated Content Removal
If a positive prediction immediately removes content, false positives can harm legitimate users. Prioritize high precision or a very low false-positive rate, route uncertain cases to review, and measure outcomes across content and user segments.
3. Fraud Review Queue
If investigators can review only a fixed number of cases, evaluate precision at the queue size, recall captured within capacity, expected prevented loss, and investigation cost. Delayed and biased labels are additional concerns because reviewed cases may be more likely to receive confirmed outcomes.
Common Follow-Up Questions
What happens when the classification threshold increases?
The model usually predicts fewer positives. TP and FP counts tend to decrease, while TN and FN counts tend to increase. Precision may improve and recall usually declines, but precision is not guaranteed to be monotonic on every finite dataset.
Can two models have the same ROC-AUC but different usefulness?
Yes. Their curves can differ in the operating region that matters, their calibration can differ, and their latency, stability, or subgroup behavior can differ. Compare them at realistic thresholds and constraints.
Does oversampling change evaluation?
Use resampling on training data when appropriate, not on the final representative test set. Evaluating on an artificially balanced test set changes prevalence-sensitive results such as precision and can misrepresent production performance.
How do you evaluate multiclass classification?
Inspect per-class results and state the averaging method. Macro averaging gives each class equal weight, weighted averaging weights by class frequency, and micro averaging aggregates decisions before calculating the metric. The choice should follow the application, especially when rare classes matter.
Which metric should you optimize during training?
The differentiable training loss and reported decision metric need not be identical. Train with an appropriate loss, compare models with representative validation data, then select the operating threshold for the product objective. Avoid repeatedly optimizing against the final test set.
A Strong 60-Second Interview Answer
I would not select a metric before defining the decision. First I would identify the positive class, class prevalence, the action caused by a prediction, and the relative costs of false positives and false negatives. Accuracy can work for balanced classes with similar costs, but under imbalance I would inspect precision and recall and compare against the prevalence baseline. For threshold-independent ranking I would consider ROC-AUC, or a precision-recall curve when positives are rare. If predicted probabilities drive decisions, I would also evaluate calibration with log loss or Brier score. Finally, I would choose the operating threshold on validation data using business costs or capacity constraints, confirm it on untouched test data, and monitor performance by segment after deployment.
That answer demonstrates definitions, trade-offs, evaluation discipline, and production judgment without pretending one metric always wins.
Practice Evaluation-Metric Interview Questions
Reading formulas creates recognition; interviews require recall and applied reasoning. On NeuraPrep, you can practice written machine learning and data science questions, ML-focused coding exercises, quizzes, and system-design scenarios, then receive AI-generated feedback informed by curated reference answers and core concepts.
Try a free question or open the NeuraPrep question bank to practice explaining metric choices under realistic follow-ups. For adjacent preparation, read the ML coding interview guide, fraud detection system design guide, and data scientist vs ML engineer comparison.