Hypothesis Testing: From Philosophy to Practice
Hypothesis testing in statistics hinges on a clear divide between objective reality and the randomness of data. A hypothesis is a statement about a fact—either true or false. Probability only enters the picture during data collection, which we assume is random.
Example: Imagine asking, "Is it summer right now?" from inside a closed room. The testing rules are set in advance. Criterion: snowfall rejects the null hypothesis H0 "It's summer." This creates a Type I error risk (α)—rejecting a true H0. The α probability is fixed upfront as the significance level.
A critical event is unlikely under a true H0. But the criterion must distinguish H0 from the alternative H1, minimizing Type II error (β) and maximizing power (1-β).
Errors and Choosing a Test
The full mantra: Reject H0 if the event is unlikely under H0 but likely under H1. H0 enjoys a presumption of innocence—it's only rejected with strong evidence (low α).
Bad criteria:
- A rare bird that only appears in summer: It would never reject a false H0 (β=1).
- A comet: Equally likely anytime, fails to distinguish hypotheses.
- Narrow temperature range of 26.0–26.1°C: Unlikely in summer but typical for it.
A good criterion (like snowfall) is unlikely under H0 and likely under H1.
H1 can be a simple negation (not summer) or specific (fall/spring). The choice affects power. In research, H0 is often "no effect" due to science's conservative nature.
Experiment Planning
Hypothesis testing is a tool for decision-making under uncertainty. Plan sample size in advance to control α and β.
Key algorithm steps:
- Formulate H0 and H1.
- Choose α (significance level).
- Define the critical event to minimize β.
- Calculate power.
- Determine sample size.
- Collect data and apply the test.
This weighs error risks against decision costs.
Key Takeaways
- H0 is protected: Rejected only by an unlikely event (α set upfront).
- Test power: Probability of detecting a false H0 (1-β).
- Randomness in data: Hypothesis is deterministic; randomness is in the sample.
- Plan ahead: Sample size and test fixed before the experiment.
- Context matters: Tailor α and H1 to the problem.
Philosophical Angles and Critiques
The Neyman-Pearson approach assumes randomness in data collection (like flipping a coin to test fairness). Critics point out: the coin's properties are fixed; probability lies in the procedure.
Alternatives like Bayesian methods directly assess hypothesis probabilities, but the classical approach focuses on errors. In IT and data science, both are used: classical for A/B tests, Bayesian for prior-driven models.
For developers: In ML, hypothesis testing validates models (H0: model no better than baseline). Do power analysis before experiments to avoid underpowered samples.
— Editorial Team
No comments yet.