Promotion Evaluation without A/B: Diff-in-Diff and PSM

Measuring Promo Impact Without A/B Testing: Diff-in-Diff & Propensity Score Matching

Open promotions without randomization require alternative evaluation methods. The main approaches are Diff-in-Diff for comparing trends and Propensity Score Matching (PSM) for constructing similar control groups. These techniques help isolate the true promo effect from seasonality and natural market trends.

A control group is essential to distinguish the promo’s impact from external factors. Without it, any metric increase during the holiday season might be mistakenly attributed to the promo—when in reality, it’s just seasonal momentum.

Diff-in-Diff: The Basic Approach with Common Control

Diff-in-Diff calculates the difference in metric trends between the target group (promo participants) and the control group (all users). Simply comparing pre- and post-promo periods ignores baseline differences—promo participants often have higher loyalty and a higher average order value from the start.

This method applies to Like-for-Like (LFL) users—those active both before and during the promo. This excludes the influence of new user acquisition.

Steps for calculation:

• For all LFL users: trend from pre-period to promo period.
• For LFL promo participants: same trend.
• The difference represents the promo effect.

Example using average order value:

| Period | Target Group | All Users |

|--------------|--------------|-----------|

| Pre-promo | ₽2,500 | ₽1,350 |

| During promo | ₽3,000 | ₽1,500 |

| Delta | +20% | +11% |

Promo effect: 20% - 11% = +9%. Incremental value: ₽3,000 / 1.09 ≈ ₽2,752 (baseline), incremental gain of ₽248.

Limitation: Promo participants may exhibit different dynamics than the general population—e.g., unique seasonal patterns.

Enhanced Diff-in-Diff: Thematic Control

For offline promos (e.g., discount on pastries), the control group consists of customers buying similar products in the same stores (other baked goods). The groups are conceptually close: baked goods sold in identical locations, one with the promo, one without.

This reduces bias but doesn’t eliminate it entirely—pastry lovers may behave differently than pizza enthusiasts (e.g., frugality vs. bulk consumption).

Propensity Score Matching: Simulating a Control Group

PSM mimics the randomization of an A/B test. Using pre-promo features (LTV, active days, average order value, account age), a model predicts the probability of participation (propensity score from 0 to 1).

The model is trained on data without the participation flag, then pairs are matched by score using k-NN. The control pool must be significantly larger than the target group for accurate matching.

Example code using XGBoost:

# Define model hyperparameters (can be optimized via Optuna)
model_params = {
    'objective': 'binary:logistic',
    'eval_metric': 'logloss',
    'learning_rate': 0.4,
    'max_depth': 3,
    'min_child_weight': 10,
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'lambda': 1.0,
    'alpha': 0.1,
    'random_state': seed,
    'n_estimators': 100
}

# Train boosting model
pm = GradientBoostedPropensityModel(early_stop=False, **model_params)
ps = pm.fit_predict(all_data[features], all_data['test'])

# Match pairs
matcher = NearestNeighborMatch(replace=False, ratio=1, random_state=seed)
all_data['score'] = ps
matched = matcher.match(all_data, 'test', ['score'])

# Final groups
matched[(matched['test'] == 0)].describe()
matched[(matched['test'] == 1)].describe()

Quality checks:

• Similarity in group metrics.
• Calibration: at score = 0.5, ~50% should be promo participants.
• Identical distribution of features across groups.

Once matched, apply Diff-in-Diff. Limitations:

• Assumes future dynamics remain similar.
• Regional seasonality effects.
• Bias from limited control pool—matching only non-participants, including those physically unable to join.

Key Takeaways

• Control group is mandatory: without it, promo effects get confounded with broader trends.
• Diff-in-Diff is simple and fast: use LFL users or thematic controls.
• PSM is more precise: requires data and computation, but closely approximates A/B testing.
• Always validate your groups: check metrics, calibration, and feature distributions.
• Prefer A/B tests when possible: avoid post-hoc evaluations if you can.

— Editorial Team