Is the difference real or due to chance?

Start here

Are you describing your patients, or drawing a conclusion about all patients?

Descriptive statistics summarise the patients you studied. Inferential statistics use those patients to draw a conclusion about all similar patients, and state how uncertain that conclusion is.

Two types of hypothesis

What exactly are you testing?

Before collecting data, you write two statements about the population. The test is carried out on the first one.

The asymmetry is deliberate. A test can show that the data are hard to explain if H₀ is true. It cannot show that H₀ is true. Fisher wrote in 1935 that "the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation." In the same way that a court verdict of "not guilty" is not a finding of innocence, "not significant" is not a finding of no difference.

In a superiority trial, H₀ is that the new treatment and the control have the same effect. In a non-inferiority trial, H₀ is that the new treatment is worse than the control by at least a pre-set margin. Trial designs are covered in the clinical trials chapter.

Write the hypotheses in symbols as well as in words. H₀ always holds the equality (=, ≤ or ≥). H₁ holds the strict inequality (≠, > or <), and its direction tells you which tail of the distribution the test uses.

For a one-tailed test, write H₁ first, because it states the direction you expect, and then write H₀ as everything else. When to use one tail is discussed in the section on one-tailed and two-tailed tests below.

Sampling distribution

If you repeated the study, how much would the result change?

Each sample gives a slightly different mean. The distribution of those means over all possible samples of the same size is the sampling distribution, and its standard deviation is the standard error (SE).

Every test and confidence interval in this chapter relies on three facts shown in the figure.

1. Sample means are centred on the population mean. With random sampling, the mean of a study does not systematically overestimate or underestimate the population mean.
2. Sample means vary far less than individual patients. SE = SD ÷ √n = 3.9 ÷ √30 = 0.71 days, less than one-fifth of the SD.
3. Sample means follow a nearly normal distribution even when the measurement is skewed, if the sample is large enough. This is the central limit theorem. The more skewed the measurement, the larger the sample needed. With a small sample of a skewed measurement, use a test that does not assume a normal distribution.

The variation between samples is called sampling error. It is not a mistake by the investigator, and a larger sample reduces it. Errors in measuring, recording or selecting patients are different, and a larger sample does not reduce them. The difference between two group means also has a sampling distribution, with its own SE, and the next section uses it. The SD and SE are compared in the descriptive statistics chapter.

How a test works

How does a test decide whether a difference is real?

It assumes there is no difference, works out how results would vary by chance alone, and then asks how unusual your result would be.

t = difference ÷ SE of the difference

difference

mean fall with Drug A minus mean fall with Drug B, here 5 mmHg

SE

standard error of the difference = SD × √(2 ÷ n) for two groups of n patients with the same SD

t

how many standard errors your result lies from 0

SE = 12 × √(2 ÷ 50) = 2.4 → t = 5 ÷ 2.4 = 2.08, df 98 → p = 0.040

df (degrees of freedom) = 50 + 50 − 2 = 98. The p-value comes from the t distribution with 98 df. Statistical software does this step.

t is a test statistic, a number calculated from the data whose distribution is known when H₀ is true. Because that distribution is known, the test can state how often a value this far from 0 would occur by chance. The four test statistics you will meet most often are listed below.

t becomes larger, and p smaller, with any of three changes to the teaching example:

• A larger difference. 10 mmHg instead of 5 gives t = 4.17 and p < 0.001.
• More patients. 100 per group instead of 50 gives SE = 1.70, t = 2.95 and p = 0.004.
• Less variation between patients. SD 6 instead of 12 gives SE = 1.2, t = 4.17 and p < 0.001.

Decide

Tests of significance: which test for which data?

A test of significance is a procedure that turns your data into a test statistic (t, χ², F) and a p-value for a stated null hypothesis. The right test depends on the type of outcome, the number of groups, whether the groups are paired, and the distribution of the data.

Parametric tests (t-test, ANOVA) assume the data follow a known distribution, usually normal. Non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis) use ranks and make fewer assumptions. The choosing a test chapter covers assumptions, checks and worked examples for each test. How to check whether data are normal enough, and what to do when they are not, is in the Practical Statistics chapter on checking assumptions.

The p-value

What does p = 0.04 tell you, and what does it not tell you?

The p-value is the probability of getting a result at least as extreme as yours if the null hypothesis were true.

p = 0.040 in the teaching example means: if Drug A and Drug B were truly equal, a difference of 5 mmHg or more in either direction would occur in about 4% of trials of this size. The data are hard to explain by chance alone, so H₀ is doubted.

It does not mean:

• a 4% chance that H₀ is true. The p-value is calculated by assuming H₀ is true, so it cannot give the probability that H₀ is true.
• a 4% chance that the result is due to chance. This is the sentence in the resident's thesis.
• that the difference is large or important. A tiny difference in a very large study gives a very small p-value.
• a 96% chance that the result will be repeated in another study.
• that, if you reject H₀, the chance your significant result is a false positive is 4% or 5%. If H₀ is actually true, rejecting it is wrong every time. α describes how often false positives occur over many studies in which H₀ is true (Greenland et al., 2016). The chance that one significant result is false also depends on how plausible the hypothesis was before the study and on the power of the study. This is the same reasoning that turns a pre-test probability into a post-test probability, covered in the Practical Statistics chapter on Bayes' rule and combining tests.

In 2016 the American Statistical Association published six principles on p-values:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Report the exact value (p = 0.040, not "p < 0.05"), together with the effect size and its 95% confidence interval.

Level of significance

Who decides that 0.05 is significant, and when?

You do, before the study starts. The level of significance (α) is the false-positive rate you are willing to accept. If p ≤ α, the result is called statistically significant.

The 0.05 level comes from Ronald Fisher's 1925 book Statistical Methods for Research Workers: "The value for which P = .05, or 1 in 20, is 1.96 or nearly 2; it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not." He called it convenient. It is a convention, not a law. Some studies use 0.01, for example when a false positive would be especially harmful.

Critical value

What is a critical value, and why do p and t point in opposite directions?

α and the critical value state the same threshold in two forms. The p-value and the test statistic state the same result in two forms. So comparing p with α always gives the same decision as comparing the test statistic with the critical value.

Significant when p ≤ α, which is the same as |test statistic| ≥ critical value

The signs point in opposite directions because a test statistic further from 0 leaves a smaller area beyond it. A large t, χ² or F goes with a small p. The rejection region (critical region) is the set of test statistic values at or beyond the critical value. If your test statistic falls in it, H₀ is rejected.

For a two-tailed t or z test, the rejection region is in both tails, so compare the absolute value of t with the critical value. χ² and F cannot be negative, and larger values mean a larger departure from H₀, so their rejection region is in the upper tail only. Teaching values at α = 0.05: the critical χ² with 1 df is 3.84, and the critical F with 2 and 27 df is 3.35.

Software prints the exact p-value, so you rarely need the critical value to reach a decision. It is still what statistical tables give, and the critical value multiplied by the SE gives the half-width of a confidence interval (1.984 × 2.4 in the teaching example).

Degrees of freedom

What are degrees of freedom, and why does the critical t depend on them?

Degrees of freedom (df) are the number of values that are free to vary once the quantities estimated from the data are fixed. For t, χ² and F tests they decide which curve, and so which critical value, is used.

The t distribution is used in place of the normal distribution when the SD is estimated from the sample, because the sample SD also varies from sample to sample. Its heavier tails allow for that extra uncertainty. As the degrees of freedom increase, t approaches the normal distribution. The critical value is 2.09 with 20 df, 1.98 with 98 df and 1.96 for the normal curve.

The sample variance divides by n − 1 instead of n for the same reason: one degree of freedom is used in estimating the mean. Worked t tests with their degrees of freedom are in the choosing a test chapter.

Interactive

Move the test statistic and watch p cross α as t crosses the critical value

α and the degrees of freedom fix the critical t. Your data fix t and the p-value. The two comparisons always give the same decision.

Comparison

One-tailed or two-tailed: which direction are you testing?

A two-tailed test looks for a difference in either direction. A one-tailed test looks in one direction only, and ignores a difference in the other direction however large it is.

Interactive

Move the observed difference and watch the one-tailed and two-tailed p-values

The p-value is the red area under the curve at and beyond your result. A two-tailed test adds the area in both directions. A one-tailed test counts only the direction named in advance.

Two types of error

In what ways can a test give the wrong answer?

The test gives a decision, and the truth is unknown. So there are four possible combinations, and two of them are errors.

For a fixed sample size, lowering α raises β. The only way to reduce both at once is to study more patients or to measure the outcome more precisely.

Interactive

Run 100 trials and count how many are significant

When no difference is real, every significant trial is a type I error. When a difference is real, the share of trials that are significant is the power, and every trial that is not significant is a type II error.

Confidence intervals

What does a 95% confidence interval actually mean?

It is a range calculated from your sample. If the same study were repeated many times, 95% of the intervals calculated this way would include the true value.

95% CI = estimate ± t × SE (t is close to 1.96 in large samples)

5 ± 1.984 × 2.4 = 0.24 to 9.76 mmHg

In the teaching example, the data are compatible with Drug A being better by as little as 0.24 mmHg or by as much as 9.76 mmHg. The interval is wide because 50 patients per group is a small study. A larger sample, a less variable outcome, or a lower confidence level (90% instead of 95%) gives a narrower interval.

For a proportion, the same method uses the SE of a proportion. Teaching example: nausea after surgery in 18 of 60 patients, p = 0.30.

95% CI = p ± 1.96 × √[p(1 − p) ÷ n] = 0.30 ± 1.96 × 0.059 = 0.18 to 0.42

This simple (Wald) interval is inaccurate when n is small or p is close to 0 or 1, and it can give limits below 0 or above 1. In those situations ask the software for the Wilson interval (0.20 to 0.43 here) or the exact Clopper-Pearson interval (0.19 to 0.43).

When no events occur, for example no wound dehiscence in 40 operations, the observed proportion is 0 and the formula gives an interval of zero width. A quick 95% upper limit is 3 ÷ n, the "rule of three" described by Hanley and Lippman-Hand in 1983: 3 ÷ 40 = 7.5%. The exact one-sided 95% upper limit is 7.2%. So no complications in 40 operations is compatible with a true rate of up to about 7%.

The confidence interval was set out by Jerzy Neyman in the 1930s. The standard error and its link with the SD are covered in the descriptive statistics chapter.

Comparison

Confidence interval or p-value: why do journals ask for both?

They are calculated from the same data and agree with each other. The confidence interval gives more information.

Two separate intervals are not a test of the difference. Teaching example with SD 12 mmHg and 50 patients per group: the mean fall is 14.0 mmHg with Drug A (95% CI 10.6 to 17.4) and 8.0 mmHg with Drug B (95% CI 4.6 to 11.4). The two intervals overlap. The difference of 6.0 mmHg has a 95% CI of 1.2 to 10.8 mmHg, which excludes 0, and p = 0.014.

Comparison

When p > 0.05, why write "H₀ not rejected" and not "H₀ accepted"?

A test can show that the data are hard to explain if H₀ is true. When p is above α, the data are compatible with H₀, but they are usually just as compatible with many other values.

Take the teaching trial of 10 patients per group with an observed difference of 10 mmHg. For H₀ "no difference", p = 0.08, so H₀ is not rejected. The 95% confidence interval is −1.3 to 21.3 mmHg. Test the same data against H₀ "the true difference is 15 mmHg" and p = 0.36. Against 20 mmHg, p = 0.08. None of these is rejected. If "not rejected" meant "accepted", you would be accepting a true difference of 0, 15 and 20 mmHg at the same time.

For a t test, the 95% confidence interval is the set of hypothesised values that a two-tailed test at α = 0.05 does not reject. This is why the interval gives more information than the words "not significant". It shows which values remain possible.

• If you see "accept H₀" in a book or a printout, read it as "H₀ not rejected".
• Write "not statistically significant", not "insignificant". A difference that is not significant can still be large enough to matter, as in this trial.
• To conclude that two treatments do not differ by an important amount, the study must be designed for it. Fix a margin in advance and show that the whole confidence interval lies inside it. This is explained in the Practical Statistics chapter on equivalence and non-inferiority tests.

Interactive

Move the hypothesised difference and watch which values the data reject

The curve gives the p-value for every hypothesised true difference. Values where the curve is above 0.05 are not rejected, and together they make up the 95% confidence interval.

Comparison

Statistically significant, but does it matter to the patient?

Statistical significance says the difference is unlikely to be due to chance alone. Clinical significance says the difference is large enough to change treatment. They answer different questions.

Multiple testing

What happens when you test many outcomes?

Each test at α = 0.05 has a 5% chance of a false positive. With many tests, the chance that at least one is falsely significant becomes large.

P(at least one false positive) = 1 − (1 − α)^k → 1 − 0.95¹⁰ = 0.401

The Bonferroni correction divides α by the number of tests, k. With 10 tests, each is judged at 0.05 ÷ 10 = 0.005. The chance of any false positive across the 10 tests then stays below 5% (4.9%). An equivalent approach multiplies each p-value by k and compares it with 0.05.

α per test = α ÷ k = 0.05 ÷ 10 = 0.005

The correction is conservative. It lowers power, especially when k is large or the outcomes are closely related. Methods that control the false discovery rate are less strict. The better solution is to name one primary outcome in the protocol, so that fewer tests are confirmatory. Other correction methods, including the tests used after ANOVA, are covered in the Practical Statistics chapter on multiple comparisons.

An example from a major trial. ISIS-2 (1988) randomised 17,187 patients with suspected myocardial infarction. Aspirin reduced the odds of vascular death by 23% overall. Among the subgroup analyses, patients born under the astrological signs of Gemini or Libra showed no apparent benefit from aspirin. This is a chance finding produced by testing many subgroups.

Interactive

Add more tests and watch the chance of a false positive rise

Each extra test at α = 0.05 adds another chance of a false positive. The Bonferroni correction judges each test at α divided by the number of tests.

By specialty

Hypothesis testing problems by specialty

Common myths

Common mistaken beliefs

• ✗Myth: p = 0.04 means a 4% chance that the null hypothesis is true.

  ✓The p-value is calculated assuming H₀ is true. It is not the probability of any hypothesis (Greenland et al., 2016).

• ✗Myth: p > 0.05 proves there is no difference.

  ✓Absence of evidence is not evidence of absence (Altman and Bland, 1995). An overview of streptokinase trials after myocardial infarction found a 22% reduction in mortality, although only 5 of the 24 trials had been significant on their own.

• ✗Myth: A smaller p-value means a bigger or more important effect.

  ✓A p-value does not measure the size of an effect or the importance of a result (ASA 2016, principle 5).

• ✗Myth: A 95% CI means a 95% probability that the true value lies in this interval.

  ✓The 95% refers to the method: 95% of intervals calculated this way over repeated studies include the true value.

• ✗Myth: 0.05 is a fixed scientific rule.

  ✓Fisher described it in 1925 as a "convenient" limit. α is chosen, and justified, before the study.

• ✗Myth: Testing many outcomes and reporting the significant ones is acceptable.

  ✓Each extra test adds a chance of a false positive. With 20 independent tests the chance of at least one is 64%.

Other ways in which correct numbers mislead, such as regression to the mean and Simpson's paradox, are covered in the Practical Statistics chapter on statistical fallacies.

Why this matters to you

Five consequences of misreading a test of significance

1. Protocol sent back

   A protocol that says "appropriate statistical tests will be applied" does not state the hypothesis, the primary outcome, α, one- or two-tailed testing, or the test.

2. A wrong conclusion in your thesis

   You write "no difference" when p = 0.20 in 30 patients. The study could not tell.

3. A wrong definition at the viva

   "p = 0.04 means a 4% chance the result is due to chance" is incorrect (ASA 2016, principle 2).

4. Misreading papers

   A significant subgroup, or one significant secondary outcome among 20, is accepted as a finding.

5. Harm to patients

   A treatment with a trivial benefit is adopted because p < 0.001, or a useful treatment is abandoned after one small trial with p > 0.05.

History

History of significance tests and confidence intervals

1. 1710

   John Arbuthnot

   Examined London parish records of births for the 82 years from 1629 to 1710. Every year, more boys than girls were born. If boys and girls were equally likely, the probability of this is (1/2)⁸², so he concluded that the pattern was not due to chance. This is the same reasoning as a significance test.

2. 1908

   William Sealy Gosset ("Student")

   A chemist in an Irish brewery, he published a method for assessing means from small samples in Biometrika under the pseudonym "Student", because his company did not allow its scientists to publish confidential data. This was the origin of Student's t-test.

3. 1925

   Ronald Fisher

   In Statistical Methods for Research Workers he wrote that P = .05 corresponds to 1.96, "or nearly 2", and that "it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not."

4. 1933

   Jerzy Neyman and Egon Pearson

   Described how to choose the most efficient test, noting that "it is very easy to control errors of the first kind": the chance of rejecting a hypothesis when it is true can be set as low as one wishes.

5. 1935

   Ronald Fisher

   In The Design of Experiments: "Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis."

6. 1937

   Jerzy Neyman

   Published "Outline of a theory of statistical estimation based on the classical theory of probability" in the Philosophical Transactions of the Royal Society, setting out the theory of confidence intervals he had worked on from 1935.

7. 1965

   Austin Bradford Hill

   Warned that "far too often we deduce 'no difference' from 'no significant difference.'"

8. 1986

   Martin Gardner and Douglas Altman

   Argued in the BMJ for confidence intervals rather than P values, and recommended them for major findings in both the main text and the abstract of a paper.

9. 1988

   ISIS-2 Collaborative Group

   A trial of 17,187 patients showed a clear benefit of aspirin after myocardial infarction. A subgroup analysis by astrological sign, in which Gemini and Libra showed no apparent benefit, is still used to show how testing many subgroups produces chance findings.

10. 1995

    Douglas Altman and Martin Bland

    Published "Absence of evidence is not evidence of absence" in the BMJ.

11. 2016

    American Statistical Association

    Issued its statement on p-values with six principles (Wasserstein and Lazar).

12. 2019

    Valentin Amrhein, Sander Greenland, Blake McShane

    With more than 800 signatories, called in Nature for the concept of statistical significance to be abandoned, and warned against concluding "no difference" because a P value is larger than a threshold.

Guidelines

Hypothesis testing in guidelines

Steps

Carrying out a test of significance, step by step

1. Write the research question and one primary outcome.
2. State H₀ (no difference) and H₁ (a difference), and decide on a one- or two-tailed test.
3. Fix α, usually 0.05 two-tailed, and write it in the protocol.
4. Choose the test that fits the type of outcome, the number of groups, pairing and the distribution.
5. Collect the data and calculate the test statistic (t, χ², F) and its degrees of freedom.
6. Find the p-value, or the critical value for your α and degrees of freedom.
7. Compare p with α, or the test statistic with the critical value. If p ≤ α, reject H₀. If p > α, do not reject H₀, and do not write that H₀ is accepted.
8. Report the effect size with its 95% confidence interval and the exact p-value, and say whether the difference matters clinically.

Reading a paper

Five checks on any study's p-values

• Were the hypothesis and the primary outcome stated in advance, in a protocol or trial registry?
• Are the effect size and its 95% confidence interval reported, and not only the p-value?
• Is the confidence interval narrow enough to confirm or rule out a difference that matters to patients?
• How many outcomes, time points and subgroups were tested, and was there any adjustment for multiplicity?
• Does the test fit the data: paired or unpaired, normal or skewed, large or small counts?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

Every TNMGRMU question on hypothesis testing

Each one is answered in this chapter. Tap a sitting to open the university's own question paper.

1. ×6Confidence intervalPh.D. RM · Jan 2025Ph.D. RM · Jul 2023MD Nuclear Medicine · May 2022MD Community Medicine · May 2019MD Nuclear Medicine · May 2018DM Cardiology · Aug 2015
2. ×5Tests of significanceMD Community Medicine · Oct 2022MD Community Medicine · Apr 2016Ph.D. RM · Jan 2016MD Community Medicine · Mar 2010MD Community Medicine · Apr 1994
3. ×2Test of significancePh.D. RM · Jan 2023Ph.D. RM · Jul 2022
4. ×2Confidence IntervalsMD Biochemistry · Nov 2021MCh Surgical Gastroenterology and Proctology · Aug 2008
5. ×2Null hypothesisMD Community Medicine · Nov 2020MD Nuclear Medicine · May 2018
6. ×2Type I and Type II errorsPh.D. RM · Jan 2018Ph.D. RM · Jan 2016
7. ×1p-value in statisticsMD Pharmacology · May 2025
8. ×1Type I and type II error in statisticsDM Virology · Feb 2025
9. ×1Null hypothesis in context of clinical trialsDM Medical Oncology · Feb 2025
10. ×1Explain the ‘p’ value in biostatistics and its role in evaluating a studyMCh Vascular Surgery · Feb 2025
11. ×1ANCOVA test of significanceMD Community Medicine · Oct 2024
12. ×1Significance of p value in hypothesis testingMD Community Medicine · May 2024
13. ×1Describe alpha error, beta error, p value and power in statisticsDM Medical Oncology · Jan 2024
14. ×1Concept of statistical significancePh.D. RM · Jul 2024
15. ×1P value does not provide a good measure of evidence –Justify the statementMD Community Medicine · Oct 2023
16. ×1Type II Error & Type I ErrorPh.D. RM · Jan 2023
17. ×1Null hypothesis and Alternate hypothesisPh.D. RM · Jul 2023
18. ×1As a researcher you are asked to identify the reasons for and factors influencing non-compliance among Diabetic (type 2). Write a detailed project proposal highlighting the research design including the plan for statistical analysisPh.D. RM · Jul 2023
19. ×1Explain the ‘Test of Significance’Ph.D. RM · Jul 2023
20. ×1Explain type-I and type II errorsPh.D. RM · Jul 2023
21. ×1Confidence interval and its importancePh.D. RM · Jul 2023
22. ×1Write a note on Odds Ratio and Confidence IntervalDM Neonatology · Aug 2022
23. ×1Describe Null hypothesis, alternate hypothesis, type 1 error, type 2 error in statisticsDM Medical Oncology · Aug 2022
24. ×1What are the Test of Significance? Describe the different tests for different types of dataPh.D. RM · Aug 2021
25. ×1Type I and Type II errorDM Medical Oncology · Feb 2020
26. ×1Tests for statistical significance with discrete dataMD Community Medicine · May 2019
27. ×1Differentiate between descriptive statistical methods and inferential statistical methodsMD Community Medicine · May 2019
28. ×1P Value and Odds ratioDM Cardiology · Feb 2019
29. ×1Explain the types of errors with examplesPh.D. RM · Jul 2018
30. ×1Explain the types of errors with examplePh.D. RM · Jul 2017
31. ×1Statistical significance of Standard deviationMD Biochemistry · Apr 2013
32. ×1P Value in Statistics- the Rationale and significance of itMD Community Medicine · Apr 2013
33. ×1Soctal security (b>) Gene therapy ic} Tests of significanceMD Community Medicine · Apr 2000
34. ×1Discuss tumour metastasis and its Mmanazyement. 25) 3+ Yrite briefly on: Tumour cell heterogeneity (b) Overcoming drug resistance (ec) Droloxifene Type I error {e) P-53DM Medical Oncology · Apr 1997
35. ×1Tests of significance in statisticsMD Community Medicine · Sep 1990

Further reading

Related Stateazy articles