AWS Machine Learning Engineer Training Course

Choosing the wrong metric is how a model with excellent numbers ends up doing the wrong thing in production. This lesson covers what each classification metric measures, when it misleads, and how to pick the decision threshold — which is a business decision the model cannot make for you.

The confusion matrix

Every classification metric derives from four counts. A true positive is a positive case correctly identified. A false positive, or type I error, is a negative case flagged as positive. A false negative, or type II error, is a positive case missed. A true negative is a negative case correctly left alone.

Before computing anything, establish which class is positive and what each error costs in the scenario. In fraud detection, a false negative is a completed fraudulent transaction and a false positive is an annoyed customer whose legitimate purchase was declined. In cancer screening, a false negative is a missed diagnosis and a false positive is an unnecessary follow-up test. Those costs are not symmetric, and they are what decide the metric.

The metrics

• Accuracy — correct predictions over all predictions. Intuitive and dangerously misleading on imbalanced data: if one class is 0.5 percent of records, predicting the majority class always achieves 99.5 percent accuracy and catches nothing.
• Precision — of the cases predicted positive, how many were positive. Answers "when the model raises an alarm, can I trust it." Optimize precision when acting on a positive is expensive or disruptive.
• Recall, also called sensitivity or the true positive rate — of the actual positives, how many were found. Answers "how much am I missing." Optimize recall when a missed positive is the costly error.
• Specificity, the true negative rate — of the actual negatives, how many were correctly left alone.
• F1 — the harmonic mean of precision and recall, a single number for when both matter. The harmonic mean punishes imbalance between the two, so a model with excellent precision and terrible recall scores poorly. F-beta generalizes it, weighting recall more heavily when beta is above one.
• Balanced accuracy — the mean of recall and specificity, a fairer summary than accuracy under imbalance.
• Matthews correlation coefficient — a correlation between predicted and actual labels that uses all four cells of the matrix and stays informative under heavy imbalance.

For multiclass problems, metrics are computed per class and then averaged. Macro averaging treats every class equally, which surfaces poor performance on rare classes. Micro averaging aggregates the counts first, which weights classes by frequency and is dominated by the common ones. Weighted averaging sits between them. Choose macro when the rare classes matter as much as the common ones.

Threshold-independent curves

Most classifiers output a score, and the label comes from comparing that score to a threshold. Two curves describe performance across all thresholds at once.

The ROC curve plots the true positive rate against the false positive rate as the threshold sweeps. The area under it, AUC-ROC, is the probability that a randomly chosen positive scores higher than a randomly chosen negative. A value of 0.5 is random, 1.0 is perfect. Its weakness is that the false positive rate has a large denominator when negatives dominate, so on severely imbalanced data AUC-ROC can look excellent while the model is impractical.

The precision-recall curve plots precision against recall. Its area, AUC-PR or average precision, is far more informative under heavy imbalance because both axes concern the positive class. The baseline is the positive class prevalence rather than 0.5. When a scenario describes rare positives, prefer PR-based evaluation.

Choosing the operating threshold

The default of 0.5 is a convention, not a recommendation. Set the threshold from the constraint the business actually has:

• Capacity constraint — a review team can process a fixed number of cases per day. Set the threshold so the volume of positives matches capacity, and report precision at that point.
• Minimum recall requirement — regulation or safety requires catching a stated share of positives. Set the threshold to meet it and accept the resulting precision.
• Expected cost minimization — assign a monetary cost to each error type, compute expected cost across the threshold sweep, and choose the minimum. This is the most defensible approach when the costs are knowable.

Two refinements matter in production. First, calibration: if downstream logic treats the score as a probability — expected loss, ranking against a monetary threshold — the scores must be calibrated, and boosted trees and neural networks often are not. Platt scaling and isotonic regression fit a correction on a held-out set. Second, the threshold is a deployment parameter, not a model parameter. Keep it configurable outside the model artifact so it can be adjusted as costs or capacity change, without retraining and redeploying.

Finally, evaluate on data that resembles production. A metric computed on a rebalanced test set overstates precision, because precision depends on prevalence. If you resampled to train, evaluate on the original class distribution.