Random Variables, Expectation and Variance

A random variable X is a function that assigns a number to each outcome of an experiment: the sum of two dice, the number of defective items in a batch, the time until a server fails.

Discrete random variables

A discrete random variable takes countably many values. Its probability mass function (pmf) is p(x) = P(X = x), with p(x) ≥ 0 and Σ p(x) = 1.

Example. Let X be the number of heads in 3 tosses of a fair coin. The 8 outcomes are equally likely:

Continuous random variables

A continuous random variable has a probability density function (pdf) f(x) ≥ 0 with ∫ f(x) dx = 1 over the whole line, and

P(a ≤ X ≤ b) = ∫ₐᵇ f(x) dx.

A single value has probability 0: P(X = a) = 0. So P(a ≤ X ≤ b) = P(a < X < b). A density can be larger than 1 (only its integral must be 1).

Example. f(x) = 2x on [0, 1] (and 0 elsewhere) is a valid pdf: ∫₀¹ 2x dx = 1. Then P(X ≤ 0.5) = ∫₀^0.5 2x dx = 0.25.

The cumulative distribution function

The cdf F(x) = P(X ≤ x) works for both kinds. It is non-decreasing, goes from 0 at −∞ to 1 at +∞, and is right-continuous.

• P(a < X ≤ b) = F(b) − F(a).
• For a continuous X, f(x) = F′(x) and F(x) = ∫ from −∞ to x of f(t) dt.
• For the example above, F(x) = x² on [0, 1].

The median m satisfies F(m) = 0.5; for f(x) = 2x, m² = 0.5 so m = 1/√2 ≈ 0.707.

Expectation

The expected value (mean) is the probability-weighted average:

• discrete: E[X] = Σ x p(x);
• continuous: E[X] = ∫ x f(x) dx.

More generally, E[g(X)] = Σ g(x) p(x) or ∫ g(x) f(x) dx (the "law of the unconscious statistician").

Rules (they hold whether or not the variables are independent):

• E[aX + b] = a E[X] + b
• E[X + Y] = E[X] + E[Y] (linearity of expectation)

For the 3 coin tosses: E[X] = 0 · 1/8 + 1 · 3/8 + 2 · 3/8 + 3 · 1/8 = 12/8 = 1.5. For f(x) = 2x on [0, 1]: E[X] = ∫₀¹ x · 2x dx = 2/3.

Indicator trick. Linearity makes many expectations easy. The expected number of fixed points of a random permutation of n items is n × (1/n) = 1, whatever n is: item i is a fixed point with probability 1/n, and we add these up, even though the events are dependent.

Variance and standard deviation

Var(X) = E[(X − μ)²] = E[X²] − (E[X])², where μ = E[X]. The standard deviation is σ = √Var(X).

• Var(aX + b) = a² Var(X): shifting does not change spread; scaling by a multiplies it by a².
• Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y), which equals Var(X) + Var(Y) when X and Y are uncorrelated (in particular, independent).
• Var(X − Y) = Var(X) + Var(Y) − 2 Cov(X, Y). Variances of independent variables add even when you subtract the variables.

For the 3 coin tosses: E[X²] = (0 + 3 + 12 + 9)/8 = 3, so Var(X) = 3 − 1.5² = 0.75. For f(x) = 2x: E[X²] = ∫₀¹ 2x³ dx = 1/2, so Var(X) = 1/2 − 4/9 = 1/18.

Joint distributions, covariance and correlation

Two discrete random variables have a joint pmf p(x, y) = P(X = x, Y = y). The marginal pmfs come from summing: p_X(x) = Σᵧ p(x, y). X and Y are independent if p(x, y) = p_X(x) p_Y(y) for every x and y. For continuous variables, the same holds with densities and integrals.

• Covariance: Cov(X, Y) = E[(X − μ_X)(Y − μ_Y)] = E[XY] − E[X]E[Y]. Cov(X, X) = Var(X), and Cov(aX + b, cY + d) = ac Cov(X, Y).
• Correlation: ρ(X, Y) = Cov(X, Y) / (σ_X σ_Y), always between −1 and 1. ρ = ±1 exactly when Y = aX + b with a ≠ 0 (a perfect linear relationship).
• If X and Y are independent, E[XY] = E[X]E[Y], so Cov(X, Y) = 0. The converse is false: X uniform on {−1, 0, 1} and Y = X² have Cov(X, Y) = E[X³] − E[X]E[X²] = 0, yet Y is completely determined by X.

Example. Joint pmf of X and Y:

E[X] = 0.7, E[Y] = 0.5, E[XY] = 0.4, so Cov(X, Y) = 0.4 − 0.35 = 0.05. They are not independent: P(X = 0, Y = 0) = 0.2, but P(X = 0) P(Y = 0) = 0.3 × 0.5 = 0.15.

Conditional distributions and conditional expectation

The conditional pmf of X given Y = y is p(x | y) = p(x, y) / p_Y(y), and the conditional expectation is E[X | Y = y] = Σ x p(x | y). In the table, E[X | Y = 1] = 1 × 0.4/0.5 = 0.8.

The law of total expectation: E[X] = E[ E[X | Y] ] = Σᵧ E[X | Y = y] P(Y = y). Here E[X | Y = 0] = 0.3/0.5 = 0.6, and 0.6 × 0.5 + 0.8 × 0.5 = 0.7 = E[X].

from fractions import Fraction as F
from itertools import product

# Number of heads in 3 fair coin tosses
pmf = {}
for t in product([0, 1], repeat=3):
    pmf[sum(t)] = pmf.get(sum(t), 0) + F(1, 8)
EX = sum(x * p for x, p in pmf.items())
EX2 = sum(x * x * p for x, p in pmf.items())
print("pmf:", {x: str(p) for x, p in sorted(pmf.items())})
print("E[X] =", EX, " Var(X) =", EX2 - EX ** 2)

# The joint table from the lesson
joint = {(0, 0): F(2, 10), (0, 1): F(1, 10), (1, 0): F(3, 10), (1, 1): F(4, 10)}
E = lambda g: sum(g(x, y) * p for (x, y), p in joint.items())
ex, ey = E(lambda x, y: x), E(lambda x, y: y)
cov = E(lambda x, y: x * y) - ex * ey
vx, vy = E(lambda x, y: x * x) - ex ** 2, E(lambda x, y: y * y) - ey ** 2
print("E[X] =", ex, " E[Y] =", ey, " Cov(X, Y) =", cov)
print(f"correlation = {float(cov) / (float(vx) * float(vy)) ** 0.5:.4f}")

# Uncorrelated but dependent: X uniform on {-1, 0, 1}, Y = X^2
xs = [-1, 0, 1]
print("Cov(X, X^2) =", sum(F(x ** 3, 3) for x in xs) - sum(F(x, 3) for x in xs) * sum(F(x * x, 3) for x in xs))

pmf: {0: '1/8', 1: '3/8', 2: '3/8', 3: '1/8'}
E[X] = 3/2  Var(X) = 3/4
E[X] = 7/10  E[Y] = 1/2  Cov(X, Y) = 1/20
correlation = 0.2182
Cov(X, X^2) = 0

Practice questions