Hypothesis Testing - z, t and Chi-Squared Tests

A hypothesis test decides whether data provides enough evidence against a default claim. Did the new drug lower blood pressure, or is the difference just chance? Is this die fair?

The framework

1. State the null hypothesis H₀, the default "no effect" claim (μ = 100, the die is fair), and the alternative H₁ (μ ≠ 100, μ > 100, the die is not fair).
2. Choose the significance level α, the risk of wrongly rejecting H₀ that we accept, usually 0.05.
3. Compute a test statistic from the data, whose distribution is known when H₀ is true.
4. Decide: reject H₀ if the statistic falls in the critical region (values unlikely under H₀), or equivalently if the p-value is at most α. Otherwise, fail to reject H₀.

"Fail to reject" is not the same as "accept": the data may simply be too limited to detect an effect.

One-tailed and two-tailed tests

• Two-tailed (H₁: μ ≠ μ₀): reject when the statistic is far out in either tail. At α = 0.05 with a z statistic, reject when |z| > 1.96.
• Right-tailed (H₁: μ > μ₀): reject when z > 1.645 at α = 0.05.
• Left-tailed (H₁: μ < μ₀): reject when z < −1.645.

The p-value

The p-value is the probability, assuming H₀ is true, of getting a test statistic at least as extreme as the one observed. A small p-value means the data would be surprising if H₀ were true. It is not the probability that H₀ is true.

For a two-tailed z-test, p = 2(1 − Φ(|z|)); for a right-tailed test, p = 1 − Φ(z).

Errors and power

For a fixed sample size, lowering α (being stricter about false alarms) raises β. Larger samples raise the power without changing α.

The z-test for a mean (σ known)

z = (X̄ − μ₀) / (σ/√n), which is N(0, 1) under H₀ (exactly for a normal population, approximately for large n).

Example. A machine is supposed to fill bottles with μ = 500 ml, with known σ = 40 ml. A sample of 64 bottles has X̄ = 510 ml. Is the machine off at α = 0.05?

• H₀: μ = 500, H₁: μ ≠ 500 (two-tailed).
• z = (510 − 500)/(40/8) = 10/5 = 2.
• |z| = 2 > 1.96, so reject H₀. The p-value is 2(1 − Φ(2)) ≈ 0.0455 < 0.05.

At α = 0.01 (critical value 2.576) we would not reject: the conclusion depends on α.

Proportions use the same idea: to test H₀: p = p₀, use z = (p̂ − p₀)/√(p₀(1 − p₀)/n). For 60 heads in 100 tosses against p₀ = 0.5, z = 0.1/0.05 = 2.

The one-sample t-test (σ unknown)

t = (X̄ − μ₀) / (S/√n), which has the t distribution with n − 1 degrees of freedom under H₀ (normal population).

Example. A tutor claims her students score more than 100 on average. A sample of 16 students has X̄ = 103 and S = 8. Test at α = 0.05.

• H₀: μ = 100, H₁: μ > 100 (right-tailed).
• t = (103 − 100)/(8/4) = 1.5, with 15 degrees of freedom.
• The critical value t_{0.05, 15} = 1.753. Since 1.5 < 1.753, fail to reject H₀: the data does not support the claim at the 5% level.

Two-sample and paired t-tests compare two means in the same way. For paired data (the same subjects before and after), take the differences and run a one-sample t-test on them.

Chi-squared tests

Chi-squared tests compare observed counts O with the expected counts E predicted by H₀:

χ² = Σ (O − E)² / E

Large values mean the data and H₀ disagree, so these tests are right-tailed. The expected count in every cell should be at least about 5 for the approximation to work.

Goodness of fit

Does a single categorical variable follow a claimed distribution? With k categories, the degrees of freedom are k − 1 (minus one more for each parameter estimated from the data).

Example. A die is rolled 60 times, giving counts 8, 12, 9, 11, 6, 14 for faces 1 to 6. Is it fair? Under H₀ each face is expected 10 times:

χ² = (4 + 4 + 1 + 1 + 16 + 16)/10 = 4.2, with 6 − 1 = 5 degrees of freedom.

The critical value χ²_{0.05, 5} = 11.070. Since 4.2 < 11.070, fail to reject: the counts are consistent with a fair die.

Independence (contingency tables)

Are two categorical variables independent? For a table with r rows and c columns, the expected count in each cell is

E = (row total × column total) / grand total

and the degrees of freedom are (r − 1)(c − 1).

Example. 100 people, classified by gender and preference:

Every expected count is 50 × 50/100 = 25, so χ² = 4 × (5²/25) = 4, with (2 − 1)(2 − 1) = 1 degree of freedom. The critical value χ²_{0.05, 1} = 3.841, and 4 > 3.841, so reject independence at the 5% level.

from math import sqrt
from statistics import NormalDist

Z = NormalDist()
z = (510 - 500) / (40 / sqrt(64))
print(f"z-test: z = {z:.2f}, two-tailed p-value = {2 * (1 - Z.cdf(abs(z))):.4f}")

t = (103 - 100) / (8 / sqrt(16))
print(f"t-test: t = {t:.2f}, critical value t(0.05, 15) = 1.753 -> {'reject' if t > 1.753 else 'fail to reject'} H0")

obs = [8, 12, 9, 11, 6, 14]
chi = sum((o - 10) ** 2 / 10 for o in obs)
print(f"goodness of fit: chi-squared = {chi:.2f}, critical 11.070 -> {'reject' if chi > 11.070 else 'fail to reject'} H0")

table = [[30, 20], [20, 30]]
rows = [sum(r) for r in table]
cols = [sum(c) for c in zip(*table)]
total = sum(rows)
chi = sum((table[i][j] - rows[i] * cols[j] / total) ** 2 / (rows[i] * cols[j] / total)
          for i in range(2) for j in range(2))
print(f"independence: chi-squared = {chi:.2f}, critical 3.841 -> {'reject' if chi > 3.841 else 'fail to reject'} H0")

z-test: z = 2.00, two-tailed p-value = 0.0455
t-test: t = 1.50, critical value t(0.05, 15) = 1.753 -> fail to reject H0
goodness of fit: chi-squared = 4.20, critical 11.070 -> fail to reject H0
independence: chi-squared = 4.00, critical 3.841 -> reject H0

Which test?

Practice questions