Remove-one analysis for KPI: breakdown of event contributions

Decomposing Individual Event Contributions to Aggregated KPIs: Remove-One Analysis

The Integrated Supplier Performance (ISP) metric aggregates Fill Rate, Quality Score, OTIF, and Lead Time with equal weights of 0.25 each. Businesses need not only the KPI value but also precise identification of which purchase order lowered it and by how much. Naively calculating metrics per individual purchase is incorrect: all metrics are built on aggregated values over a data window—the current date minus 14 days plus 30 days back to account for delays.

Remove-one analysis solves this: compute the metric with all data (Metric_all), then without one record (Metric_without_i), and the difference gives the record's contribution. If the metric increases after removal, that purchase worsened it.

Example Calculation for Fill Rate

Consider three purchase orders:

| Purchase | Ordered | SupplierCancels |

|----------|---------|-----------------|

| A | 100 | 5 |

| B | 200 | 0 |

| C | 50 | 15 |

Overall Fill Rate = (1 - 20/350) × 100 = 94.29%.

Without C: (1 - 5/300) × 100 = 98.33%. Contribution of C: 94.29 - 98.33 = -4.04 percentage points.

Fill Rate formula: (1 - ΣSupplierCancels / ΣOrdered) × 100. Removal changes the aggregate nonlinearly due to the denominator.

Challenges with Quality Score

Quality Score = 100 - DefectRate. DefectRate has two branches:

• DefectRate_pcs = ΣDefectivePcs / ΣOrderedPcs
• DefectRate_rub = ΣDefectiveRub / ΣOrderedRub

Selection logic:

• If (100 - DefectRate_pcs) < 90, use pcs.
• Else if (100 - DefectRate_rub) < 90, use rub.
• Otherwise, use pcs.

Problem: with remove-one, the branch might switch. For example, all data uses pcs, but without i, it uses rub. Comparing incomparable values distorts the contribution.

Solution: fix the branch at the aggregated metric level. Algorithm:

• Compute QualityScore_all and calculation_mode.
• For each purchase, calculate without_i in the same mode.

Implementation Pitfalls

• Nonlinearity: Contribution depends on the entire dataset; removing one element has a disproportionate effect.
• Formula switching: Requires fixing the mode.
• Sliding windows: Daily recalculations change contributions over time.

An alternative is what-if analysis: replace the purchase status with an ideal one and recalculate. Difference:

• Remove-one: actual contribution to the current KPI.
• What-if: hypothetical improvement with a perfect purchase.

We chose remove-one for accurately explaining actual declines.

Key Takeaways

• Aggregated metrics require remove-one for correct contribution decomposition.
• Fixing calculation_mode prevents distortions from nonlinear branches.
• Data window (14+30 days) accounts for delivery delays.
• The approach is similar to leave-one-out in ML and Shapley value in economics.
• Backend implementation combines discrete mathematics and engineering trade-offs.

Applications in Other Fields

Sensitivity analysis with remove-one is used in:

• Leave-one-out validation in machine learning.
• Explainable AI for model interpretation.
• Shapley value for contribution distribution in cooperative games.
• Data analytics for event attribution.

In backend, this becomes a task of stable calculation over large data volumes without losing interpretability.

— Editorial Team