Diversity of errors beats quality in ML ensembles

Diversity of Errors in Ensembles: Why Weak Models Outperform Strong Ones

In ensembles, the key isn't the average quality of individual models—it's how uncorrelated their errors are. An experiment on financial time series showed that a pool of models with infrequent retraining (Step 20) achieved an MAE of 8.5, beating the 9.5 from models retrained frequently (Step 1), even though individual metrics were worse (median MAE 15.7 vs. 11.6).

Experiment Methodology

Data: Time series of Black Sea FOB wheat prices, plus fundamental and macroeconomic factors. Forecast horizon: 1–12 weeks.

Base Models: 18 algorithms (Holt-Winters, Prophet, Random Forest, Ridge, KNN, Gradient Boosting) with hyperparameter variations and training periods.

Evaluation Metrics: MAE, MAPE, trend direction accuracy.

Ensembling: Lasso regression.

Key Factor: Comparing pools with different retraining frequencies.

• Step 1: Retrain on every observation.
• Step 20: Retrain every 20 periods.

Results by Metric

The Step 20 ensemble cuts MAE by 46% (from 15.7 to 8.5), while Step 1 only manages 18% (from 11.6 to 9.5). Individually, Step 1 is better: median trend accuracy 55% vs. 52.5%.

MAE Summary:
Step 1 (individual): median 11.6
Step 20 (individual): median 15.7
Step 1 Ensemble: 9.5
Step 20 Ensemble: 8.5

Step 20 leads across all horizons (1–12 weeks). The gap widens on medium- and long-term forecasts: the ensemble hits 65.7% trend accuracy, topping Step 1's best models.

Error Correlation Structure

Frequent updates (Step 1) make models converge to similar local minima, with highly correlated errors. Averaging those forecasts doesn't fix the mistakes.

Infrequent updates (Step 20) cause jumps in parameter space, creating diverse minima. Errors are weakly correlated and cancel each other out.

Scatter plots of model trajectories show the shift: Step 20 ensembles migrate to low MAE/MAPE zones, even if individual models are scattered.

Gains from Ensembling

Comparing pool vs. ensemble metrics confirms the effect: Lasso on Step 20 boosts diversity for superior performance.

Practical Tips for Building Ensembles

• Don't select solely on individual metrics: weak models add value with uncorrelated errors.
• Use greedy selection: add models that maximize gains to the current ensemble metric.
• Monitor error correlations: averaging highly correlated errors yields zero benefit.
• Vary retraining frequency: infrequent cycles increase diversity.
• Test on validation: treat error diversity as a standalone target metric.

Key Takeaways

• Error diversity trumps individual model quality in ensembles.
• Infrequent model updates generate uncorrelated errors, boosting MAE forecasts by 46%.
• The effect grows on longer horizons, where error compensation is crucial.
• Greedy selection by ensemble contribution beats sorting by solo metrics.
• Error correlation is the top indicator of pool potential.

This experiment proves: picking the 'best' models intuitively can hurt your ensemble. Focusing on error structure builds robust forecasting systems.

— Editorial Team