VWE and CUPED in A/B tests: variance reduction

VWE Complements CUPED: Reducing Variance in A/B Tests

A/B tests often fail to detect effects due to high noise in metrics. Increasing traffic or experiment duration isn't the only solution. Reducing variance estimation through CUPED or VWE can boost sensitivity. These methods address different issues: CUPED uses pre-experiment covariates, while VWE tackles variance heterogeneity across units.

Simulations show that combining methods yields the greatest benefit in scenarios with noisy users. Results are reproducible with 4000 units, 300 simulations, seed=42.

The CUPED Mechanism: Correlation as the Key to Noise Reduction

CUPED adjusts metric Y based on pre-period covariate X:

Y_cuped = Y - θ (X - E[X]), where θ = cov(Y, X) / var(X)

The effect depends on the correlation ρ between X and Y. With ρ≈0, the gain is minimal (0.2% variance reduction). At ρ=0.6, it's 40%; at ρ=0.95, it's 90%.

Simulation 1: Comparing standard deviation of effect estimation for plain diff and CUPED with ρ from 0 to 0.95. CUPED is effective only with a strong covariate-outcome link.

Common mistakes:

• Weak correlation between pre-metric and Y.
• Mixing pre- and post-treatment data.
• Ignoring variance heterogeneity across units.

VWE: Tackling Unstable Units

VWE applies weights inverse to unit variance, estimated from pre-observations. Useful when power users have high within-unit variability.

Simulation 2: 88% stable users (sd=1), 12% noisy users (sd=5). Comparing RMSE (effect=0):

| Method | RMSE | SD Estimate |

|-------------|------|------------|

| Plain diff | 0.073| 0.073 |

| CUPED | 0.064| 0.064 |

| VWE | 0.061| 0.061 |

| CUPED+VWE | 0.042| 0.042 |

The combination reduces RMSE by 42% compared to plain diff. Effective sample size is 2550 out of 4000 units due to weight redistribution.

When to Use Each Method

Prioritize CUPED

• High-quality pre-treatment covariate.
• Strong correlation with the metric.
• Noise from inter-user level variation.

Use VWE as a Supplement

• Repeated pre-observations are available for variance estimation.
• A small user group dominates variance.
• CUPED has been applied, but sensitivity remains insufficient.

Key Takeaways

• CUPED combats predictable noise through correlated covariates, with gain proportional to ρ.
• VWE minimizes the contribution of unstable units by estimating variance from pre-data.
• Combination creates synergy in heterogeneous data: RMSE drops by 42% in the simulation.
• Data validation is essential: synthetic data illustrates, but production requires analyzing noise structure.
• Effective sample size shrinks with VWE—account for this in planning.

Limitations of the Approaches

Simulations simplify reality:

• VWE here is basic, without production optimizations.
• Focus on variance/RMSE, not bias or implementation cost.
• Results aren't universal—they depend on metric variance distribution.

In production, monitor weights, robustness to outliers, and impact on business metrics.

— Editorial Team