Is the difference real?

Think like a court

A new wheat variety yielded 35 kg/da more than the old one in a trial. Does the difference come from the new variety really being better, or from natural differences between plots, in other words from chance?

A hypothesis test approaches this question like a court. The accused is innocent until proven guilty. In a test, likewise, the claim “there is no difference” is accepted until shown otherwise:

• Null hypothesis ($H_0$): there is no difference. The true means of the varieties are equal; the difference you see is due to chance.
• Alternative hypothesis ($H_1$): there is a difference.

If the data show a difference too large to be consistent with “no difference”, $H_0$ is rejected. If the evidence is insufficient, $H_0$ cannot be rejected. That does not mean $H_0$ is true. Just as an acquittal doesn’t mean “definitely innocent”, it only means “not enough evidence to show a difference”.

A test takes five steps

1. State the hypotheses. $H_0: \mu_1 = \mu_2$, $H_1: \mu_1 \neq \mu_2$.
2. Choose the significance level ($\alpha$). Usually 5%. This means keeping the risk of “declaring a difference when there is none” at 5% at most.
3. Calculate the test statistic. A number measuring how large the observed difference is compared with the variability expected by chance: t, χ² or F.
4. Compare it with the table value. If the calculated value exceeds the critical value, you are in the rejection region.
5. Make the decision and state what it means in practice. “H₀ was rejected” is not enough: write something like “the new variety yielded significantly more than the old one at the 5% level”.

A decision can go wrong in two ways

	Really no difference ($H_0$ true)	Really a difference ($H_0$ false)
H₀ rejected	Type I error (probability $\alpha$). You think an ineffective fertilizer works and recommend it to farmers.	Correct decision
H₀ not rejected	Correct decision	Type II error. You miss a genuinely better variety.

Making $\alpha$ smaller (choosing 1%) reduces Type I errors but raises the risk of missing real differences. The way to reduce both is to collect more data with more replicates.

What a p-value does and doesn’t tell you

p-value: the probability of seeing a difference at least this large by chance alone, if there were really no difference. If p = 0.02, in a world with no difference you would see this result in only 1 trial out of 50. A result that rare is good reason to doubt the claim of “no difference”.

If the p-value is below α, $H_0$ is rejected. This gives the same decision as comparing with the table value; it just carries more information.

Two-tailed or one-tailed?

If you are only asking “is there a difference?”, you use a two-tailed test: the difference could go either way. If the question had a direction before the trial was set up (“does this fertilizer increase yield?”), a one-tailed test can be used. Choosing the direction after looking at the data is cheating, because it artificially halves the p-value.

Which test?

Question	Data	Test
Does a group’s mean differ from a given value?	Measurements	One-sample t test
Do the means of two independent groups differ?	Measurements	Independent two-sample t test
Do two measurements on the same units differ?	Paired measurements	Paired t test
Do the means of three or more groups differ?	Measurements	Analysis of variance
Do counts fit the expected ratios?	Counts	Chi-square goodness-of-fit test
Are two classification traits related?	Counts	Chi-square test of independence
Do two measurements change together?	Pairs of measurements	Correlation

t tests

The logic of a t test is always the same: observed difference ÷ variability expected by chance. As the numerator grows or the denominator shrinks, t grows and the difference becomes harder to explain by chance. The denominator (the standard error) shrinks in two ways: with less variable data or with more observations.

Why is the paired test more powerful? When you compare two halves of the same plot, differences in soil, slope and moisture between plots affect both measurements the same way. Taking differences removes that variability from the calculation, and the fertilizer’s effect shows more clearly. In the “Paired” example below the differences are only 2–5 cm, but all in the same direction, so t comes out very large.

Test type

One sampleTwo independent samplesPaired

Question: Do the means of two independent groups (two varieties, two regions) differ?
Example: Two wheat varieties were each tested in eight plots (kg/da). Do their yields differ?

Variety A520, 548, 495, 562, 530, 541, 508, 553

n = 8, mean 532.13

Variety B488, 512, 470, 505, 497, 480, 515, 492

n = 8, mean 494.88

Significance level (α)5% (0.05)1% (0.01)10% (0.10)Alternative hypothesis1st ≠ 2nd (different)1st > 2nd1st < 2ndVariancesAssume equalNot equal (Welch)

$H_0: \mu_1 = \mu_2$$H_1: \mu_1 \neq \mu_2$

The red area is the rejection region (α = 0.05). The green line is the value calculated from your data.

H₀ rejected: the difference is significant

The difference between the two group means (37.25) is statistically significant. It cannot be explained by chance.

p-value = 0.0021 (significant at 1%, **)

How was it calculated?

$t_{calc}$ = 3.768

1. Compute the mean and variance of each group.

   $$\bar{x}_1 = 532.125,\quad s_1^2 = 538.6964,\quad n_1 = 8$$
2. $$\bar{x}_2 = 494.875,\quad s_2^2 = 242.9821,\quad n_2 = 8$$

3. Since the variances are assumed equal, the two variances are pooled.

   $$s_p^2 = \dfrac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2} = \dfrac{7\cdot 538.6964 + 7\cdot 242.9821}{14} = 390.8393$$

4. Standard error of the difference:

   $$S_{\bar{x}_1-\bar{x}_2} = \sqrt{s_p^2\left(\dfrac{1}{n_1}+\dfrac{1}{n_2}\right)} = 9.8848$$

5. How many standard errors is the difference between the two means?

   $$t = \dfrac{\bar{x}_1 - \bar{x}_2}{S_{\bar{x}_1-\bar{x}_2}} = \dfrac{532.125 - 494.875}{9.8848} = 3.768$$

6. Degrees of freedom n₁ + n₂ − 2 = 14.

7. Compare the calculated value with the table (critical) value.

   $$|t_{calc}| = 3.768 > t_{tab} = 2.145$$

8. The calculated value is in the rejection region: H₀ is rejected. The p-value (0.0021) is smaller than α = 0.05.

Confidence interval

A hypothesis test answers “is there a difference?” with yes or no. A confidence interval answers “in what range does the true value most probably lie?”, which is usually more informative. In a one-sample test, if the 95% confidence interval does not contain the tested value ($\mu_0$), a two-tailed test at the 5% level also rejects $H_0$. The two methods reach the same conclusion by different routes.

Chi-square tests

When the data are counts rather than measurements (how many plants got sick, how many seeds were yellow), means can’t be compared. Instead, the observed counts are compared with the expected counts:

$$\chi^2 = \sum \frac{(O - E)^2}{E}$$

The larger the gap between observed (O) and expected (E) in each class, the larger $\chi^2$. Because the difference is squared, its direction doesn’t matter; that is why the chi-square test is always right-tailed.

• Goodness-of-fit test: do the counts fit the ratios a theory predicts? Mendel’s 9:3:3:1 ratio is the classic example, and plant breeders still use it to check segregation ratios.
• Test of independence: do two classification traits affect each other? Does the disease rate vary by variety?

Test type

Goodness of fitTest of independence

Significance level (α)5% (0.05)1% (0.01)10% (0.10)

Question: Do the observed counts fit the ratios the theory expects?
Example: Mendel’s pea experiment: counts of round-yellow, round-green, wrinkled-yellow and wrinkled-green seeds. Inheritance theory predicts a 9:3:3:1 ratio. Do the data fit?

Class	Observed (O)	Expected ratio	Expected (E)	Delete
			312.75	×
			104.25	×
			104.25	×
			34.75	×

Add class

$H_0: \text{The observed counts fit the expected ratios}$$H_1: \text{They do not fit}$

The red area is the rejection region (α = 0.05). The green line is the value calculated from your data.

H₀ not rejected: the difference is not significant

The gap between observed and expected counts can be explained by chance; the data fit the ratios.

p-value = 0.9254

How was it calculated?

$\chi^2_{calc}$ = 0.47

1. Expected frequencies: the total count (556) is shared out according to the ratios.

   $$E_i = n \cdot \dfrac{\text{ratio}_i}{\sum \text{ratio}} \Rightarrow 312.75,\ 104.25,\ 104.25,\ 34.75$$

2. In each class, square the gap between observed and expected, divide by expected and add up.

   $$\chi^2 = \sum \dfrac{(O - E)^2}{E} = \dfrac{(315 - 312.75)^2}{312.75} + \dfrac{(108 - 104.25)^2}{104.25} + \dfrac{(101 - 104.25)^2}{104.25} + \dfrac{(32 - 34.75)^2}{34.75}$$
3. $$\chi^2 = 0.016 + 0.135 + 0.101 + 0.218 = 0.47$$

4. Degrees of freedom = number of classes − 1 = 3. The chi-square test is always right-tailed: only large deviations indicate a poor fit.

5. Compare the calculated value with the table (critical) value.

   $$\chi^2_{calc} = 0.47 \le \chi^2_{tab} = 7.815$$

6. The calculated value is in the acceptance region: H₀ cannot be rejected. The p-value (0.9254) is not smaller than α = 0.05.