1. Overview: Foundations of Regression Evaluation

In regression and forecasting tasks, models output continuous values (e.g., house prices, temperature, sales volume) rather than discrete classes. The core approach to evaluating regression models is: compare predicted values against true values, then summarize overall performance with a scalar metric^[1].

1.1 Three Metrics at a Glance

2. MAE (Mean Absolute Error)

MAE is the arithmetic mean of absolute errors across all samples—the most intuitive regression metric^[2].

2.1 Intuitive Understanding

MAE answers: “On average, how far off is each prediction?” Its value shares the target variable’s unit, directly interpretable as “average deviation of X units.” For example, MAE = 50K for house price prediction means the model averages 50K off from true prices^[2].

2.2 Key Characteristics

3. RMSE (Root Mean Squared Error)

RMSE squares errors, takes the mean, then takes the square root—restoring the unit to match the target variable^[4].

3.1 Intuitive Understanding

RMSE answers: “How severe are the large errors?” By squaring before averaging, RMSE applies quadratic penalty—a sample with error=10 contributes 100, while error=1 contributes only 1, a 100:1 ratio (vs. MAE’s 10:1)^[4].

3.2 Key Characteristics

3.3 Mathematical Proof: RMSE ≥ MAE

E[|X|]² ≤ E[X²]
i.e., MAE² ≤ MSE = RMSE²
∴ RMSE ≥ MAE

4. WMAPE (Weighted Mean Absolute Percentage Error)

WMAPE is an improved version of MAPE that divides total absolute error by total actual values, yielding a unitless percentage^[5].

4.1 Key Difference from MAPE

Traditional MAPE computes per-sample percentage errors then averages:

MAPE = (1/n) × Σ(|yᵢ - ŷᵢ| / |yᵢ|) × 100%

MAPE’s fatal flaw: when yᵢ is small (near 0), a single sample’s percentage error approaches infinity, severely distorting the overall metric. WMAPE avoids this by using “total error ÷ total actual”^[5].

4.2 Key Characteristics

5. Three-Metric Comparison

5.1 Numerical Example

Sample   Actual y    Pred ŷ    Error e    |e|    e²
  1       100K       95K        5K       5      25
  2       200K      210K      -10K      10     100
  3       150K      145K        5K       5       25
  4        80K       90K      -10K      10     100
  5       300K      280K       20K      20     400
  ─────────────────────────────────────────────────
  Total   830K      820K        —       50     650

6. Error Distribution & Optimal Metrics

Choosing MAE vs. RMSE isn’t subjective preference—it’s determined by the statistical distribution of errors. There’s rigorous theoretical backing^[7].

6.1 Two Classic Error Distributions

6.2 Maximum Likelihood Perspective

7. Outlier Diagnostics

The RMSE/MAE ratio is a powerful tool for diagnosing the degree of outlier influence in data^[4].

8. Application Scenario Guide

8.1 Selection Decision Flow

9. Python Code Examples

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error

# True values and predictions
y_true = np.array([100, 200, 150, 80, 300])
y_pred = np.array([95, 210, 145, 90, 280])

# ── MAE ──
mae = mean_absolute_error(y_true, y_pred)
print(f"MAE:   {mae:.2f}")
# MAE:   10.00

# ── RMSE ──
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
print(f"RMSE:  {rmse:.2f}")
# RMSE:  11.40

# ── WMAPE ──
wmape = np.sum(np.abs(y_true - y_pred)) / np.sum(np.abs(y_true)) * 100
print(f"WMAPE: {wmape:.2f}%")
# WMAPE: 6.02%

# ── Diagnostic: RMSE/MAE ratio ──
ratio = rmse / mae
print(f"RMSE/MAE ratio: {ratio:.2f}")
# RMSE/MAE ratio: 1.14  (uniform errors, no significant outliers)

# ── Comparison: with an outlier ──
y_true_out = np.array([100, 200, 150, 80, 300])
y_pred_out = np.array([95, 210, 145, 90, 100])  # Sample 5 off by 200

mae_out = mean_absolute_error(y_true_out, y_pred_out)
rmse_out = np.sqrt(mean_squared_error(y_true_out, y_pred_out))
wmape_out = np.sum(np.abs(y_true_out - y_pred_out)) / np.sum(y_true_out) * 100

print(f"\n--- With Outlier ---")
print(f"MAE:   {mae_out:.2f}")     # MAE:   42.00
print(f"RMSE:  {rmse_out:.2f}")    # RMSE:  89.44  ← squared amplification
print(f"WMAPE: {wmape_out:.2f}%")  # WMAPE: 25.30%
print(f"Ratio: {rmse_out/mae_out:.2f}")  # Ratio: 2.13  ← outlier present!

# ── Custom WMAPE function (handles zeros) ──
def wmape(y_true, y_pred):
    """Calculate WMAPE, handling zero-sum case"""
    total_actual = np.sum(np.abs(y_true))
    if total_actual == 0:
        return np.nan
    return np.sum(np.abs(y_true - y_pred)) / total_actual * 100

# ── Cross-validation with multiple metrics ──
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestRegressor

model = RandomForestRegressor(random_state=42)
mae_scores = cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
rmse_scores = cross_val_score(model, X, y, cv=5, scoring="neg_root_mean_squared_error")

print(f"CV MAE:  {-mae_scores.mean():.2f} ± {mae_scores.std():.2f}")
print(f"CV RMSE: {-rmse_scores.mean():.2f} ± {rmse_scores.std():.2f}")

10. Common Pitfalls & Best Practices