Moment Generating Function: formulas and properties

Moment-Generating Functions in Probability: Computation and Properties

The moment-generating function (MGF) defines a random variable's distribution through the expected value of an exponential: $M_W(t) = E[e^{tW}]$. This method lets you extract all moments of the distribution by differentiating at $t=0$. The MGF exists if the integral converges in a neighborhood around zero, ruling out heavy-tailed distributions like the Cauchy.

Moments $E[W^r]$ describe the shape of the distribution. The first moment is the mean $E[W]$, and the second relates to variance $Var(W) = E[W^2] - (E[W])^2$. Direct computation of higher moments is tricky due to tail amplification, but the MGF handles this via Taylor series expansion.

Mathematical Expansion and Moment Extraction

Expand $e^{tW}$ in a Taylor series:

$$e^{tW} = 1 + tW + \frac{(tW)^2}{2!} + \frac{(tW)^3}{3!} + \dots$$

Taking expectations yields:

$$M_W(t) = 1 + t E[W] + \frac{t^2 E[W^2]}{2!} + \frac{t^3 E[W^3]}{3!} + \dots$$

Differentiating with respect to $t$ and setting $t=0$ isolates the moments:

• $M_W'(0) = E[W]$
• $M_W''(0) = E[W^2]$
• In general: $M_W^{(r)}(0) = E[W^r]$

This works because the coefficient of $t^r / r!$ equals $E[W^r]$. For the MGF to exist, $E[e^{tW}] < \infty$ must hold in some neighborhood around $t=0$.

Additivity Property for Independent Variables

A key advantage of MGFs is their multiplicativity for sums of independent random variables $V$ and $W$:

$$M_{V+W}(t) = M_V(t) \cdot M_W(t)$$

Proof:

$$M_{V+W}(t) = E[e^{t(V+W)}] = E[e^{tV} e^{tW}] = E[e^{tV}] E[e^{tW}] = M_V(t) M_W(t)$$

This simplifies convolution analysis, where direct computation is cumbersome.

Example: Exponential Distribution

Consider $W \sim Exp(\lambda)$ with density $f(x) = \lambda e^{-\lambda x}$, $x \geq 0$. The MGF is:

$$M_W(t) = E[e^{tW}] = \int_0^\infty e^{tx} \lambda e^{-\lambda x} dx = \lambda \int_0^\infty e^{-(\lambda - t)x} dx$$

The integral converges for $t < \lambda$ and equals $\frac{\lambda}{\lambda - t}$:

$$M_W(t) = \frac{\lambda}{\lambda - t}, \quad t < \lambda$$

Derivatives:

$$M_W'(t) = \frac{\lambda}{(\lambda - t)^2} \implies M_W'(0) = \frac{1}{\lambda}$$

$$M_W''(t) = \frac{2\lambda}{(\lambda - t)^3} \implies M_W''(0) = \frac{2}{\lambda^2}$$

Variance: $Var(W) = \frac{2}{\lambda^2} - \left( \frac{1}{\lambda} \right)^2 = \frac{1}{\lambda^2}$.

Practical Applications in Analysis

MGFs are used to identify distributions, test hypotheses, and run simulations. In data science and machine learning, they aid in analyzing sums of model errors or generating synthetic data. Comparing MGFs across distributions streamlines convergence proofs.

For instance, the normal distribution $N(\mu, \sigma^2)$ has:

$$M(t) = \exp\left( \mu t + \frac{\sigma^2 t^2}{2} \right)$$

This makes it easy to verify properties of sums of normals.

Key Takeaways:

• MGFs encode all moments via derivatives at $t=0$.
• Multiplicative for independent sums, simplifying convolutions.
• Don't exist for heavy-tailed distributions (Cauchy).
• Convergence domain defines the $t$ range.
• Ideal for analytical computation of higher-order moments.

— Editorial Team