1. Overview: Why Evaluation Metrics Matter

In machine learning classification tasks, model predictions need objective, quantifiable metrics to assess quality. Knowing “the model got 95 correct” is insufficient—we need to understand its performance on positive and negative classes separately, whether it’s biased, and whether it meets specific business requirements^[1].

1.1 Four Core Metrics at a Glance

2. Confusion Matrix: The Foundation

The Confusion Matrix is a 2×2 table that cross-tabulates model predictions against ground truth labels, producing four fundamental quantities. All evaluation metrics are derived from these four^[2].

2.1 The Four Basic Quantities

2.2 Memory Aid

3. Accuracy

Accuracy is the most intuitive evaluation metric—the proportion of all samples that the model predicted correctly^[3].

3.1 Intuitive Understanding

Accuracy answers: “How does the model perform overall?” It treats correct predictions of positive and negative classes equally in the numerator, and is the only metric that considers TN.

3.2 The Fatal Pitfall of Accuracy

4. Precision

Precision measures: of all samples predicted as positive, what fraction is actually positive^[1].

4.1 Intuitive Understanding

Precision answers: “When the model says ‘yes’, how trustworthy is it?” It focuses on the purity of positive predictions.

4.2 Precision Characteristics

5. Recall

Recall (also known as Sensitivity, Hit Rate, or True Positive Rate) measures: of all actual positive samples, what fraction was correctly predicted as positive^[4].

5.1 Intuitive Understanding

Recall answers: “Of all actual positives, how many did the model find?” It focuses on coverage—did we miss any positives?

5.2 Recall Characteristics

6. F1-score

F1-score is the harmonic mean of Precision and Recall, designed to balance the two^[5].

Equivalent form:

F1 = 2 × TP / (2 × TP + FP + FN)

6.1 Why Harmonic Mean?

The key difference between harmonic and arithmetic mean: harmonic mean is more sensitive to extreme values. If Precision=0.01 and Recall=1.0, the arithmetic mean = 0.505 (looks okay), but the harmonic mean F1 = 0.0198 (exposes the low Precision)^[5].

6.2 F-beta Generalization

F1 is a special case of the more general F-β metric (β=1). β controls Recall’s weight relative to Precision:

7. Relationships and Trade-offs

7.1 The Precision-Recall Seesaw

There is a natural trade-off between Precision and Recall. By adjusting the classification threshold, you can improve one metric but typically at the expense of the other^[4].

7.2 Four-Metric Comparison

7.3 Numerical Example: Multi-dimensional Evaluation

8. Imbalanced Dataset Challenges

When positive/negative ratios are extreme (fraud 1:1000, rare disease 1:10000), Accuracy fails. We need Precision, Recall, and F1^[6].

8.1 Accuracy Failure Demo

8.2 Strategies for Imbalanced Data

1. Choose the Right Metric Use Precision, Recall, F1 instead of Accuracy. Consider PR-AUC (area under Precision-Recall curve).
2. Resampling Oversample minority class (SMOTE) or undersample majority class to balance training data.
3. Class Weighting Assign higher weight to minority class in loss function to make the model care more about its errors.
4. Threshold Tuning Default 0.5 threshold may not be optimal; adjust based on business needs to balance P/R.
5. Use Appropriate Algorithms Ensemble methods (XGBoost) and anomaly detection approaches are more robust to imbalance.

9. Multi-class Extensions

In multi-class tasks (e.g., digit recognition 0-9), the confusion matrix expands to N×N. How do we extend binary metrics to multi-class? Three common averaging strategies^[7]:

9.1 Three Averaging Strategies

9.2 Macro vs Micro Example

Per-class Recall:
  Class A (100 samples): Recall = 90/100 = 0.90
  Class B ( 50 samples): Recall = 40/50  = 0.80
  Class C ( 10 samples): Recall = 5/10   = 0.50

Macro-Recall = (0.90 + 0.80 + 0.50) / 3 = 0.733
  → Each class equally important; C's low Recall drags down average

Micro-Recall = (90+40+5) / (100+50+10) = 135/160 = 0.844
  → Sample-weighted; A's high proportion boosts the average

Weighted-Recall = (0.90×100 + 0.80×50 + 0.50×10) / 160 = 0.844
  → Same as Micro (because of Recall's weighted nature)

9.3 One-vs-Rest Approach

Multi-class metrics are typically computed using the One-vs-Rest (OvR) strategy: treat each class as “positive” in turn, all others as “negative,” compute that class’s P/R/F1, then aggregate using one of the averaging strategies above.

10. Practical Guide & Code Examples

10.1 Scenario-Metric Matching Guide

10.2 Decision Flowchart

10.3 Python Code Example

# Computing classification metrics with scikit-learn
from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    classification_report,
    confusion_matrix
)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# 1. Generate imbalanced data (1:9 ratio)
X, y = make_classification(
    n_samples=1000,
    weights=[0.1],
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# 2. Train model
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# 3. Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(f"Confusion Matrix:\n{cm}")
# [[256  11]
#  [  13  20]]

tn, fp, fn, tp = cm.ravel()
print(f"TP={tp}, TN={tn}, FP={fp}, FN={fn}")
# TP=20, TN=256, FP=11, FN=13

# 4. Four core metrics
print(f"Accuracy:  {accuracy_score(y_test, y_pred):.4f}")
# Accuracy:  0.9200

print(f"Precision: {precision_score(y_test, y_pred):.4f}")
# Precision: 0.6452

print(f"Recall:    {recall_score(y_test, y_pred):.4f}")
# Recall:    0.6061

print(f"F1-score:  {f1_score(y_test, y_pred):.4f}")
# F1-score:  0.6250

# 5. Full classification report (with Macro/Micro/Weighted)
print(classification_report(y_test, y_pred, target_names=["Negative", "Positive"]))
#               precision  recall  f1-score  support
#     Negative       0.95     0.96      0.95      267
#     Positive       0.65     0.61      0.63       33
#    accuracy                           0.92      300
#   macro avg       0.80     0.78      0.79      300
# weighted avg       0.92     0.92      0.92      300

# 6. Cross-validation with average strategy
from sklearn.model_selection import cross_val_score

macro_f1 = cross_val_score(model, X, y, cv=5, scoring="f1_macro")
weighted_f1 = cross_val_score(model, X, y, cv=5, scoring="f1_weighted")
print(f"Macro F1 CV:    {macro_f1.mean():.4f}")
print(f"Weighted F1 CV:{weighted_f1.mean():.4f}")

10.4 Common Pitfalls