Why do correct numbers still mislead?

Start here

How can a study with correct arithmetic reach the wrong conclusion?

A statistical fallacy is an error in reasoning from data. The calculations can be correct while the sample, the comparison, the analysis, the presentation or the interpretation leads the reader to a false conclusion.

Who was studied

Can a sample give the wrong answer even when it is large?

Yes. A sample that differs systematically from the population gives a biased answer, and a larger biased sample only makes that answer more precise.

• Survivors and prevalent cases: patients who died early, or recovered quickly, never enter a study of existing cases.
• Volunteers: people who agree to take part differ in health, motivation and adherence from those who do not.
• Hospital and referral samples: tertiary centres see more severe and unusual cases. Results may not apply to patients in primary care.
• Too small a sample: a small study has low power, so it misses real effects. When it does find a significant result, the estimated effect tends to be too large. A review of neuroscience studies by Button and colleagues (2013) estimated a median statistical power of 21%.

Sampling methods and selection bias are covered in the bias and validity chapter.

What was compared

Were the groups alike in everything except the treatment?

A difference between groups can come from the treatment, or from any other difference between the patients in the groups.

• Case-mix: one hospital, clinic or treatment group has sicker, older or more complex patients.
• Different follow-up or exposure time: rates compared over unequal periods. Use events per person-time, not events per person.
• Different intensity of looking: a group that is examined more often has more of the outcome recorded.
• Different adherence or dropout: patients who stay on a treatment differ from those who stop.
• Chance imbalance: even randomisation can leave small trials unbalanced on an important prognostic factor. Look at the baseline table.

The remedy is to compare like with like: randomisation, matching, stratified results, or adjustment in a regression model for the factors that differ. Confounding is explained in the bias and validity chapter.

Mixing distinct groups

Can combining two groups reverse the result seen in each of them?

Yes. When a factor that affects the outcome is distributed very differently between the groups being compared, the pooled comparison can point the opposite way from every stratum. This is Simpson's paradox.

Made-up teaching example. Hospital A, a referral centre, treats 400 severe and 100 mild patients. Hospital B treats 100 severe and 400 mild patients. Hospital A has the higher success rate in both strata.

Pooled, Hospital B looks better, because most of its patients are mild and mild patients do well everywhere. Adjusted for severity, Hospital A is better: standardised to the same case-mix (half mild, half severe), A has 82.5% success and B 75.0%.

The reversal was pointed out by Pearson (1899) and Yule (1903). Simpson's 1951 paper showed how combining contingency tables can lead to paradoxical conclusions. The fix is to report stratum-specific results, or an estimate adjusted for the factor, such as the Mantel-Haenszel odds ratio.

Interactive

Change the case-mix of the two hospitals and watch the pooled comparison reverse

Selection on an extreme value

Why do patients selected for a very high reading improve without treatment?

Every measurement is the patient's usual level plus some chance variation. Patients picked because a reading was extreme are, on average, patients whose chance variation was high that day. At the next reading, the chance part is new, so the average moves back towards the usual level.

expected visit 2 = population mean + R × (visit 1 − population mean)

R

reliability = variance of true values ÷ variance of observed values = SD_true² ÷ (SD_true² + SD_error²)

R = 1

no measurement error: no regression to the mean

R = 0.5

half the observed spread is chance: the expected second reading is halfway back to the mean

Teaching example: true systolic pressure averages 130 mmHg with SD 12, and visit-to-visit variation has SD 8, so R = 144 ÷ 208 = 0.69. Patients reading 160 or more at visit 1 average 165.3 mmHg. Their expected visit 2 value is 130 + 0.69 × 35.3 = 154.4 mmHg, a fall of 10.8 mmHg with no treatment. The name comes from Galton's 1886 paper on “regression towards mediocrity” in the heights of parents and children.

Barnett and colleagues (2005) noted that the effect becomes more noticeable as measurement error increases and when follow-up is examined only in a subgroup selected on a baseline value. How to deal with it: include a randomised control group selected in the same way, select patients on the average of two or more readings, take a fresh baseline reading after selection, and analyse follow-up values adjusted for baseline (analysis of covariance).

Interactive

Recall patients with a high reading and watch their average fall with no treatment

Level of the data

Does an association between districts hold for the people in them?

Not necessarily. An association found with group-level data (districts, countries, years) may be weaker, absent or reversed among individuals. Assuming it holds for individuals is the ecological fallacy.

• Why it happens: group averages hide the variation within groups, and factors that differ between groups (age, income, diagnosis rates) are mixed in with the exposure. The data cannot show whether the exposed individuals are the ones with the outcome.
• Example: Messerli (2012) reported a correlation of r = 0.791 (p < 0.0001) between chocolate consumption per person and Nobel laureates per 10 million people across 23 countries. Country averages cannot show that the people who eat chocolate are the ones who receive Nobel prizes, and Messerli himself noted that the direction could be reversed.
• Classic reference: W. S. Robinson, “Ecological correlations and the behavior of individuals” (1950).
• Remedy: collect individual-level data, or analyse individuals and groups together in a multilevel model. Use ecological studies to generate hypotheses and to study exposures that act on whole areas.

Ecological study design is described in the study designs chapter.

Summaries

When do percentages and means mislead?

A percentage hides its denominator, and a mean hides the spread and the shape of the data.

A published example of confusing the two units: Gigerenzer and colleagues (2007) pointed out that Naylor and colleagues (1992) described a fall in myocardial infarction from 3.9% to 2.5% as “a 1.4% decrease” instead of a decrease of 1.4 percentage points.

Interpretation

What can a P-value not tell you?

A P-value is the probability of data at least as extreme as those observed if the null hypothesis were true. It is not the probability that the hypothesis is true, and it does not measure the size or importance of an effect.

Significance is not importance

With 20,000 patients per group, a difference of 1.0 percentage point in mortality (15.0% vs 16.0%) gives p = 0.006, although the whole 95% CI (−1.7 to −0.3 points) lies within the 3 points that matter. Large samples make small differences significant. Judge importance from the effect size and its confidence interval, compared with the smallest difference that matters to patients.

“No evidence of effect” is not “evidence of no effect”

Trial A (6/40 vs 8/40 deaths) gives p = 0.56, but its 95% CI runs from 21.6 points fewer deaths to 11.6 points more. The study cannot separate a large benefit from a large harm. Only Trial B, whose interval lies within ±3 points, supports the statement that there is no important difference. To show that two treatments are equivalent, a study must be designed for that purpose.

p-hacking and multiple testing

Every extra outcome, time point, subgroup, cut-off, covariate set or interim look gives another chance of a false positive. If 20 independent outcomes are tested at α = 0.05 when no effect exists, the chance of at least one significant result is 1 − 0.95²⁰ = 64%. Simmons and colleagues (2011) showed with simulations and experiments that undisclosed flexibility in collecting, analysing and reporting data greatly increases the actual false-positive rate. Trying analyses until one reaches p < 0.05 and reporting only that one is called p-hacking.

The American Statistical Association's 2016 statement on P-values, type I and II errors, and multiple testing are covered in the hypothesis testing chapter.

Interpretation

When does an association mean that one thing causes the other?

An association can come from chance, bias, confounding by a third factor, or reverse causation, as well as from cause and effect.

• Chance: check the confidence interval and how many associations were tested.
• Bias: check how patients were selected and how exposure and outcome were measured.
• Confounding: a third factor linked to both. Grey hair and cataract are associated because both increase with age.
• Reverse causation: the outcome, or its early stage, causes the exposure. People who feel unwell may stop exercising before a diagnosis is made.

In 1965 Austin Bradford Hill listed nine viewpoints from which to study an association “before we cry causation”: strength, consistency, specificity, temporality, biological gradient, plausibility, coherence, experiment and analogy. He stated that they are not hard-and-fast rules. Whatever else is uncertain, a cause must come before its effect. A randomised trial removes confounding by design, which is why it gives the strongest evidence that a treatment causes an effect.

Time

Why do patients who received a treatment later seem to live longer?

To be counted as treated, a patient must survive until treatment starts. If that waiting time is credited to the treated group, the treated group gets survival time during which it could not die. This is immortal time bias.

• First described: in the 1970s, in cohort studies of the survival benefit of heart transplantation (Suissa, 2008). Gail's 1972 reassessment asked “Does cardiac transplantation prolong life?”
• Oscar winners: a 2001 study reported that Academy Award winners lived 3.9 years longer than other performers (79.7 vs 75.8 years). A reanalysis that did not credit winners' years before winning found an advantage closer to 1 year, and not statistically significant (Sylvestre and colleagues, 2006).
• Survivorship bias is the wider problem: analysing only people who survived long enough to be observed.
• How to avoid it: treat the exposure as time-varying, so that follow-up before treatment starts counts as untreated time, or start follow-up for everyone at a fixed landmark time after diagnosis and classify patients by treatment received by then.

Interactive

Delay the start of a drug that does nothing and watch the naive analysis show a benefit

Presentation

How can correct results be reported in a misleading way?

The same result can be made to look large or small by the measure chosen, the axis drawn and the outcomes selected for the abstract.

Relative risk without absolute risk

• A relative change describes the risk ratio. An absolute change describes how many patients in 100 or 1,000 are affected. Report both, with the baseline risk.
• Naylor and colleagues (1992) randomised 100 Toronto clinicians to see trial results as relative or absolute risk reductions. Ratings of effectiveness were lower when the same data were shown as absolute differences (P < 0.001).
• CONSORT 2025 item 26 asks for “presentation of both absolute and relative effect size” for binary outcomes. RR, ARR and NNT are explained in the risk and survival chapter.

Graphs that exaggerate

A bar chart whose vertical axis does not start at zero makes small differences look large. The lie factor and other misleading graphs are shown in the data presentation chapter.

Selective reporting

Reporting only the outcomes, time points or subgroups that reached significance makes chance findings look like results. Compare the paper with its protocol or trial registration. CONSORT 2025 item 28 asks that ancillary analyses distinguish pre-specified from post hoc.

Decide

Which fallacy might explain this result?

Start from what the paper claims.

Comparison

No evidence of effect, or evidence of no effect?

Comparison

Regression to the mean, or a real treatment effect?

The formula

Worked examples: adjusting, predicting and splitting time

Made-up teaching data, calculated in Python (SciPy and statsmodels).

1. Simpson's paradox: crude and Mantel-Haenszel odds ratios

Odds of success in Hospital A compared with Hospital B, from the table in the Simpson's paradox section.

crude OR = (375 × 80) ÷ (125 × 420) = 0.57

OR_MH = Σ(a_id_i ÷ n_i) ÷ Σ(b_ic_i ÷ n_i)

a, b

successes and failures in Hospital A within a stratum

c, d

successes and failures in Hospital B within the same stratum

n

patients in the stratum

1. Mild stratum: a = 95, b = 5, c = 360, d = 40, n = 500. ad ÷ n = 3,800 ÷ 500 = 7.6. bc ÷ n = 1,800 ÷ 500 = 3.6. Stratum OR 2.11.
2. Severe stratum: a = 280, b = 120, c = 60, d = 40, n = 500. ad ÷ n = 22.4. bc ÷ n = 14.4. Stratum OR 1.56.
3. OR_MH = (7.6 + 22.4) ÷ (3.6 + 14.4) = 30 ÷ 18 = 1.67 (95% CI 1.11 to 2.50, Cochran-Mantel-Haenszel p = 0.014).
4. The crude comparison (OR 0.57, chi-square p < 0.001) points the opposite way from both strata and from the adjusted estimate.

2. Regression to the mean: the expected second reading

1. Observed SD = √(12² + 8²) = √208 = 14.42 mmHg. Reliability R = 144 ÷ 208 = 0.692.
2. Cut-off 160 mmHg is z = 30 ÷ 14.42 = 2.08 above the mean. 1.9% of patients are recalled.
3. Mean visit 1 reading in those recalled = 130 + 14.42 × φ(2.08) ÷ [1 − Φ(2.08)] = 165.3 mmHg.
4. Expected visit 2 = 130 + 0.692 × 35.3 = 154.4 mmHg. Expected fall without treatment: 10.8 mmHg.

3. Immortal time: naive and time-split rates

200 patients with the same constant death rate, 0.10 per patient-month, are followed for 12 months. 100 are planned to start a drug with no effect at month 3. 74 of them are still alive then and count as treated.

1. Naive analysis, all follow-up of the 74 counted as treated: 44 deaths in 662 patient-months, 6.6 per 100 patient-months. Never treated (126 patients): 96 deaths in 736 patient-months, 13.0 per 100. Rate ratio 0.51.
2. Time-split analysis, the first 3 months of the 74 counted as untreated: treated 44 deaths in 440 patient-months, 10.0 per 100. Untreated 96 deaths in 958 patient-months, 10.0 per 100. Rate ratio 1.00.

In a real paper

Immortal time bias and Simpson's paradox in published studies

Software

How to run it

Checking for a fallacy usually means running the analysis that removes it: stratifying, adjusting for baseline, correcting for multiple tests, or splitting follow-up time.

Stratified analysis (Simpson's paradox, confounding)

• SPSS: Analyze > Descriptive Statistics > Crosstabs. Put the treatment or hospital in Rows, the outcome in Columns, and the stratifying factor in Layer. Under Statistics, tick Cochran's and Mantel-Haenszel statistics, which are computed once for all layers.
• jamovi: the Contingency Tables analysis (the same analysis as jmv::contTables) accepts layer variables and shows a table for each stratum. Its Mantel-Haenszel option is a test for trend, not the stratified odds ratio, so use R or SPSS for the adjusted odds ratio.

# R: Cochran-Mantel-Haenszel test and common odds ratio
tab <- array(c(95, 5, 360, 40,      # mild: A success, A failure, B success, B failure
               280, 120, 60, 40),   # severe
             dim = c(2, 2, 2),
             dimnames = list(outcome  = c("success", "failure"),
                             hospital = c("A", "B"),
                             severity = c("mild", "severe")))
mantelhaen.test(tab)                # the last dimension holds the strata

Regression to the mean (baseline adjustment)

• SPSS: Analyze > General Linear Model > Univariate. Follow-up value as Dependent Variable, group as Fixed Factor, baseline value as Covariate.
• jamovi: ANCOVA, with the follow-up value as the dependent variable, group as a factor and baseline as a covariate (jmv::ancova).
• R: lm(followup ~ group + baseline, data = d)

Multiple testing

p <- c(0.004, 0.012, 0.03, 0.04, 0.20)
p.adjust(p, method = "holm")   # controls the family-wise error rate
p.adjust(p, method = "BH")     # controls the false discovery rate

Immortal time (time-varying treatment)

library(survival)
# pts: one row per patient: id, futime (months), died (0/1)
# trt: one row per treated patient: id, start (months), on_drug = 1
base <- tmerge(pts, pts, id = id, death = event(futime, died))
tv   <- tmerge(base, trt, id = id, drug = tdc(start, on_drug, 0))
coxph(Surv(tstart, tstop, death) ~ drug, data = tv)

Checked against the documentation: mantelhaen.test and p.adjust (R stats), tmerge with tdc(y, x, init) and event() (survival 3.8-12), contTables and ancova (jmv 2.8.0), and the IBM SPSS Statistics 25 pages for Crosstabs Statistics and GLM Univariate.

Reporting

How to report it

Show the reader the analysis that answers the fallacy, not only the one that gives the smallest P-value.

• “Success rates are reported within severity strata. The odds of success were higher in Hospital A in both mild (OR 2.11) and severe patients (OR 1.56). The Mantel-Haenszel odds ratio adjusted for severity was 1.67 (95% CI 1.11 to 2.50). The crude odds ratio was 0.57 because the two hospitals treated very different proportions of severe patients.”
• “Because patients were enrolled on a single raised reading, follow-up blood pressure was compared between randomised groups with analysis of covariance adjusted for the baseline value.”
• “Treatment was analysed as a time-varying exposure from the day it started, so that follow-up before treatment was counted as untreated time.”
• “Mortality was 15.0% with the new treatment and 15.5% with standard care (difference −0.5 percentage points, 95% CI −2.7 to 1.7). The confidence interval excludes the pre-specified smallest important difference of 3 percentage points.”
• For binary outcomes give absolute and relative effect sizes (CONSORT 2025, item 26). State which analyses were pre-specified (item 28).

Reading a paper

A checklist for reading an article's statistics

Read the paper in the order the study was done: who, compared with whom, measured how, analysed how, and concluded what.

By specialty

Statistical fallacies by specialty

Common myths

Common mistaken beliefs

• ✗Myth: A large database cannot give a biased answer.

  ✓Size reduces random error, not bias. A large sample of survivors, volunteers or referral patients gives a precise but biased estimate.

• ✗Myth: If a group improves from baseline, the treatment worked.

  ✓Patients selected for a high value improve on average without treatment. Only a comparable control group shows the treatment effect.

• ✗Myth: A result seen in the combined data holds in each subgroup.

  ✓Simpson's paradox: the pooled comparison can reverse the comparison in every stratum.

• ✗Myth: A correlation between countries shows the same relation in people.

  ✓Group-level associations may be absent or reversed among individuals: the ecological fallacy.

• ✗Myth: p > 0.05 shows that two treatments are the same.

  ✓It shows that the study could not detect a difference. Look at the confidence interval, or at a trial designed to show equivalence.

• ✗Myth: p = 0.03 means a 3% chance that the result is a false positive.

  ✓ASA 2016, principle 2: P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

• ✗Myth: Patients who received the drug lived longer, so the drug prolongs life.

  ✓If treatment started after follow-up began, treated patients had to survive to receive it. Check for immortal time bias.

Why this matters to you

Five consequences of missing a statistical fallacy

1. A thesis conclusion that does not hold

   A before-after study of patients recruited at their worst reports an improvement that is mostly regression to the mean.

2. A wrong ranking of hospitals or units

   Crude outcome rates penalise the unit that accepts the sickest patients.

3. A common viva error

   "The difference was not significant, so the two drugs are equally effective."

4. Misreading papers at journal club

   Observational studies with immortal time or ecological data are presented as proof of benefit.

5. Harm to patients

   A relative risk headline leads patients to stop a useful treatment, or a trivial benefit is adopted because p was small.

History

History of statistical fallacies in medicine

1. 1886

   Francis Galton

   Published "Regression towards mediocrity in hereditary stature", the origin of the term regression.

2. 1903

   G. Udny Yule

   Published "Notes on the theory of association of attributes in statistics", one of the first descriptions of the reversal later named after Simpson.

3. 1950

   W. S. Robinson

   Published "Ecological correlations and the behavior of individuals" in the American Sociological Review.

4. 1951

   E. H. Simpson

   Published "The interpretation of interaction in contingency tables", showing how combining contingency tables can lead to paradoxical conclusions.

5. 1965

   Austin Bradford Hill

   Set out nine viewpoints for judging whether an association is causal, and warned against treating them as hard-and-fast rules.

6. 1972

   Mitchell Gail

   Published "Does cardiac transplantation prolong life? A reassessment". Immortal time bias was first identified in studies of heart transplantation in the 1970s.

7. 1992

   David Naylor and colleagues

   Showed that clinicians rated the same trial results as more effective when presented as relative rather than absolute risk reductions.

8. 1995

   Douglas Altman and Martin Bland

   Published "Absence of evidence is not evidence of absence" in the BMJ.

9. 2006

   Marie-Pierre Sylvestre, Ella Huszti and James Hanley

   Reanalysed the 2001 report that Oscar winners live 3.9 years longer. Without immortal time the advantage was closer to 1 year and not significant.

10. 2008

    Samy Suissa

    Described immortal time bias in pharmaco-epidemiology and showed that the bias grows with the duration of immortal time.

11. 2011

    Joseph Simmons, Leif Nelson and Uri Simonsohn

    Showed that flexibility in data collection, analysis and reporting greatly increases false-positive rates.

12. 2013

    Katherine Button and colleagues

    Estimated a median statistical power of 21% in neuroscience studies, with overestimated effects and low reproducibility as consequences.

13. 2016

    American Statistical Association

    Released a statement on statistical significance and P-values with six principles.

Guidelines

Fallacies in guidelines and statements

Reading a paper

Five checks for statistical fallacies in any paper

• Who was included, and who could not have been (people who died, did not attend, or were never referred)?
• Were patients selected on an extreme value, and is there a control group selected the same way?
• Do the groups differ in severity or other risk factors, and do stratified or adjusted results agree with the crude ones?
• Did follow-up start at the same time for treated and untreated patients?
• Are effects given as absolute and relative differences with 95% confidence intervals, compared with a difference that matters?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

TNMGRMU questions on statistical fallacies

Questions from MD, MS, DM, MCh and Ph.D. papers that this chapter helps you answer. Tap a sitting to open the university's own question paper.

1. ×1Describe ecological fallacyMD Community Medicine · Oct 2024
2. ×1How will you study statistical fallacies in an article?MD Community Medicine · Aug 2020

Further reading

Related Stateazy articles