Why do studies get the wrong answer?

Two types of error

Why does a study give a different answer from the truth?

A result can differ from the true value in two ways. Random error scatters results on both sides of the truth. Systematic error, called bias, shifts results in one direction.

A third reason for a wrong answer is confounding, covered below. The sampling chapter shows sampling error and bias in survey samples.

Errors other than chance variation between samples are called non-sampling errors. The most common is non-response. It does two kinds of harm. Fewer people answer, so the estimate is less precise, and the people who do not answer usually differ from those who do, so the estimate is biased. A larger sample corrects only the first. For the second, make repeated efforts to reach a random subsample of non-responders and compare them with responders. Record a non-response as missing, separately from a true zero and from "not applicable".

Where bias enters

At which stage of a study can bias enter?

Bias is a systematic error in the design, conduct, analysis or reporting of a study that moves the result away from the true value. Name each bias by the stage at which it enters.

Selection bias

Who got into your study, and who did not?

Selection bias occurs when the people in a study differ systematically from the population you want to describe, so the estimate or the association is wrong.

Common types of selection bias

• Berkson's bias (admission rate bias): an exposure-disease association in hospital patients is distorted because both affect the chance of admission. Take controls from the general population, or use new cases.
• Healthy worker effect: people must be healthy enough to be employed, so a workforce has lower mortality than the general population, which includes people too ill to work (McMichael, 1976). Compare workers with other workers.
• Volunteer bias: volunteers differ from the population of interest, in socio-demographic factors and in their attitudes to the study.
• Non-response bias: people who do not respond differ from those who do in their exposures or outcomes.
• Attrition bias (loss to follow-up): unequal losses from the study groups change the characteristics of the groups.
• Prevalence-incidence (Neyman) bias: a study of existing cases misses people who died quickly or recovered. Use new (incident) cases.
• Healthy user bias: people who take up and continue a preventive treatment tend to be healthier in other ways, such as exercising, not smoking and attending health checks.

Prevent selection bias at the design stage. Define the population, use probability or consecutive sampling, take cases and controls from the same source population, and keep response and follow-up high. Analysis cannot reliably correct it.

Information bias

Was information collected the same way in every group?

Information bias arises from systematic differences in how information is collected, recalled, recorded or handled.

• Recall bias: people remember past exposures with different accuracy or completeness. Parents of a child with a serious illness may search their memory harder than parents of a healthy child.
• Interviewer bias: an interviewer who knows who is a case asks or probes cases and controls differently. Use a standard questionnaire, neutral wording, and interviewers who do not know the group or the hypothesis.
• Observer bias: the person recording an observation, such as a blood pressure or a scan finding, records it with a systematic error, often shaped by what they expect. Use observers who do not know the group.
• Misclassification: people are put in the wrong exposure or outcome category. If the errors are similar in all groups (non-differential), the association tends to be underestimated. If they differ between groups (differential), it can be overestimated or underestimated.
• Detection bias: the groups differ in how outcomes are looked for or confirmed.
• Hawthorne effect: people change their behaviour because they know they are being observed.

To prevent information bias, blind the people who measure outcomes, use standard tools and written protocols, prefer records made before the outcome was known, and train observers.

Regression to the mean

Why do patients chosen for an extreme reading improve without treatment?

A single reading is the true value plus measurement error. People selected because their reading was extreme include many whose error pushed the reading up, so their next reading is, on average, closer to the population mean.

Expected second reading = mean + r × (first reading − mean)

r = 10² ÷ (10² + 8²) = 0.6098, so 130 + 0.6098 × 25.50 = 145.55

mean

population mean of the readings

r

reliability of one reading: true variance ÷ (true variance + error variance)

Barnett and colleagues (2005) described regression to the mean as a statistical phenomenon that can make natural variation in repeated data look like real change. Its effect is larger when measurement error is larger, and when follow-up measurements are examined only in a subgroup selected on a baseline value. A study that enrols patients because of a high reading and reports the fall at the next visit will show improvement even with a useless treatment.

• Use a randomised control group. Both groups regress by the same amount, so the difference between them estimates the effect of the treatment.
• Average several baseline readings. With two readings averaged, r rises to 0.76 and the expected fall in the same example is 6.0 mmHg instead of 10.0.
• Select on one reading and use a later reading as the baseline. The measurement error that led to selection then does not enter the change.

Improvement through the natural course of a self-limiting illness is a different reason for change without treatment. A randomised control group deals with both.

Screening

Why do screen-detected cancers seem to survive longer?

Survival is counted from the date of diagnosis. Screening moves that date earlier and finds more slow-growing disease, so survival looks longer even if no death is delayed.

Overdiagnosis is the detection of disease that would never have caused symptoms or death in the person's lifetime. These people are counted as survivors and raise survival figures further. People who attend screening also tend to be healthier than people who do not, which is a form of selection bias.

A screening programme is tested fairly by a randomised trial that compares deaths from the disease in the group invited to screening with the control group.

Reporting

Why do published studies look more positive than all the studies that were done?

Publication bias occurs when the chance of a study being published depends on its results.

In a follow-up of research projects approved by an Oxford ethics committee, 285 had been analysed. Studies with statistically significant results were more likely to be published than studies that found no difference (adjusted odds ratio 2.32) (Easterbrook and colleagues, 1991).

Related reporting biases include outcome reporting bias (only some of the prespecified outcomes are reported, depending on the results) and language bias (a review that reads only English-language studies can reach a biased answer). Trial registration, a published protocol, and searches of trial registries and non-English sources reduce them.

Other sources of error

Which other errors can bias a study?

Bias also enters through how a study is run, recorded and analysed. Each error below has a practical safeguard.

Suissa (2008) defined immortal time as a span of cohort follow-up during which, because of the way exposure is defined, the outcome under study could not occur. The bias was first identified in the 1970s in cohort studies of survival after heart transplantation. It has since appeared in observational studies that reported various medicines to be extremely effective.

Errors of analysis and interpretation, such as testing many outcomes and reporting only the significant ones, removing inconvenient data, or reading a correlation as a cause, are covered in the Practical Statistics chapter on statistical fallacies.

Confounding

Is a third factor mixing into the comparison?

Confounding occurs when the effect of the exposure is mixed with the effect of a third factor that is linked to both the exposure and the outcome.

Three conditions for a confounder

1. It is associated with the exposure (smokers drink more coffee).
2. It is a risk factor for the outcome, including in people who are not exposed (smoking causes MI in non-coffee drinkers).
3. It is not on the causal pathway: the exposure does not cause it.

How to control confounding

OR_MH = Σ(a × d ÷ n) ÷ Σ(b × c ÷ n)

a, b

exposed cases and exposed controls in one stratum

c, d

unexposed cases and unexposed controls in the same stratum

n

total number of people in that stratum

(5.33 + 7.2) ÷ (5.33 + 7.2) = 12.53 ÷ 12.53 = 1.0

Smokers (n = 150): 80 × 10 ÷ 150 = 5.33 and 40 × 20 ÷ 150 = 5.33. Non-smokers (n = 500): 10 × 360 ÷ 500 = 7.2 and 40 × 90 ÷ 500 = 7.2. Only measured confounders can be adjusted for. Randomisation is the only method that also deals with unmeasured ones.

With many confounders, a propensity score, the predicted probability of being exposed given the measured confounders, lets exposed and unexposed people with similar scores be compared. It is described in the Practical Statistics chapter on propensity scores. Like regression, it adjusts only for confounders that were measured.

Confounding by indication is common in studies of treatments. The condition for which a drug is prescribed, not the drug, causes the outcome. Sicker patients receive the treatment, and the treatment appears harmful.

Interactive

Change how smoking is linked to coffee and to MI, and compare the crude and Mantel-Haenszel odds ratios

A third factor confounds only when it is linked to both the exposure and the outcome. Stratifying by the factor shows whether the crude odds ratio was distorted.

Effect modification

Or is the effect really different in different groups?

Effect modification is present when the strength of an association differs across levels of a third factor. It is a real finding to report, not an error to remove.

Decide

Is the third factor a confounder, a mediator or an effect modifier?

Answer these questions about the factor.

Interactive

Read a study and name the bias

Ask at which stage the error entered: choosing participants, measuring, analysing or reporting.

Comparison

Is the result correct for the patients studied, and does it apply to others?

Internal validity asks whether the result is correct for the people studied. External validity asks whether it applies to other people and settings.

Randomisation, allocation concealment and blinding are described in detail in the clinical trials chapter.

Association or causation

When does an association show cause and effect?

After chance, bias and confounding have been considered, Hill (1965) set out nine viewpoints to weigh before deciding that an association is causal.

1. Strength: how large the association is.
2. Consistency: whether it has been seen repeatedly, by different people, in different places, circumstances and times.
3. Specificity: whether the exposure is linked to a particular disease and not to many.
4. Temporality: whether the exposure came before the disease.
5. Biological gradient: whether more exposure goes with more disease, a dose-response relation.
6. Plausibility: whether a biological explanation exists. Hill noted that this depends on the knowledge of the day and cannot be demanded.
7. Coherence: whether the association fits what is known of the natural history and biology of the disease.
8. Experiment: whether preventive action taken because of the association does in fact prevent the disease.
9. Analogy: whether a similar exposure is known to cause a similar effect.

Hill wrote that none of the nine viewpoints can bring indisputable evidence for or against cause and effect, and none is required in every case. They help to answer one question: is there any other way of explaining the facts that is as likely, or more likely, than cause and effect?

Validity of a measurement tool

Does your questionnaire or test measure what it is meant to measure?

A measurement tool is valid if it measures the concept it is meant to measure. Each type of validity is a different kind of evidence for this.

A clinical score or prediction model needs similar evidence. Check its discrimination, how well it separates patients with and without the outcome (the C statistic, which is the area under the ROC curve), and its calibration, how closely predicted risks match observed risks. A score usually performs less well in new patients than in the sample it was developed in, so test it in a separate validation sample, or correct for this optimism. The Practical Statistics chapter on bootstrap validation shows how.

Reliability

Does your tool give the same result when the measurement is repeated?

A tool is reliable if repeated measurements under the same conditions give similar results. Reliability reflects random error. A tool can be reliable and still not valid.

κ = (p_o − p_e) ÷ (1 − p_e)

p o

observed proportion of agreement

p e

agreement expected by chance, from each rater's totals

Teaching example: two radiologists read the same 100 chest films. Both call 40 abnormal and 45 normal. Radiologist A alone calls 5 abnormal, and radiologist B alone calls 10 abnormal. A calls 45% abnormal and B calls 50%.

p_o = 85 ÷ 100 = 0.85, p_e = 0.45 × 0.50 + 0.55 × 0.50 = 0.50

κ = 0.35 ÷ 0.50 = 0.70

Kappa is 1 when agreement is perfect and 0 when agreement is no better than chance. For two methods that measure the same continuous quantity, a high correlation does not show that they agree. Use the graphical method for agreement described by Bland and Altman (1986).

The intraclass correlation coefficient has several forms, so a report should state which one was used. Koo and Li (2016) suggested reading the 95% confidence interval of an ICC as poor reliability below 0.5, moderate from 0.5 to 0.75, good from 0.75 to 0.9, and excellent above 0.90. Kappa and the ICC are worked through in the Practical Statistics chapters on kappa and agreement and on the ICC and Bland-Altman analysis.

Interactive

Type counts into an agreement table and watch kappa change

Kappa compares the agreement observed with the agreement expected by chance from the totals of each rater.

Internal consistency

Do the items of a scale measure the same thing?

When a questionnaire adds several items into one score, the items should agree with each other. Internal consistency measures how far they do.

Split-half reliability divides the items into two halves, for example odd and even items, and correlates the two half-scores. Each half is only half as long as the full scale, so the correlation is corrected with the Spearman-Brown formula.

Full-scale reliability = 2r ÷ (1 + r) = 2 × 0.60 ÷ 1.60 = 0.75

Cronbach's alpha uses every item instead of one split. In its standardised form it depends only on the number of items and the average correlation between them.

α = k × r̄ ÷ [1 + (k − 1) × r̄]

k

number of items

r̄

average correlation between pairs of items

• Adding items raises alpha even when the items correlate no more strongly, so a long scale can have a high alpha without being a better scale.
• Tavakol and Dennick (2011) noted that a high alpha, above 0.90, may mean that some items are redundant, asking the same question in a different form.
• Alpha does not show that the items measure a single concept. If a questionnaire has several subscales, calculate alpha for each subscale (Tavakol and Dennick, 2011).
• For items answered yes or no, the equivalent coefficient is the Kuder-Richardson formula 20 (KR-20).

Teaching numbers, calculated in Python. More on the reliability and validity of scales is in the Practical Statistics chapter on reliability and validity of scales.

Comparison

Accuracy or precision: what is the difference?

Accuracy is how close a measurement is to the true value. Precision is how close repeated measurements are to each other.

Coefficient of variation (CV) = SD ÷ mean × 100

Teaching example from a laboratory: repeated glucose measurements have a mean of 100 mg/dL and an SD of 4, so CV = 4.0%. Repeated creatinine measurements have a mean of 1.0 mg/dL and an SD of 0.1, so CV = 10.0%. The SD of glucose looks larger, but creatinine is the less precise test. The SD is in the units of each test and depends on the level measured. The CV is a percentage of the mean, so it compares tests and levels directly.

Interactive

Change the systematic and random error and watch accuracy and precision change

Systematic error moves every reading in one direction. Random error scatters readings on both sides.

Radiology

Why do experienced radiologists miss or misread findings?

Reading an image means searching it and then deciding what the findings mean. Busby and colleagues (2018) reviewed the cognitive biases that affect both steps, with ways to reduce each.

• Satisfaction of search: less attention to further abnormalities after the first one is found. Complete a systematic second search.
• Satisfaction of report (alliterative error): repeating the impression in a previous report, one's own or a colleague's. Form your impression before reading the earlier report.
• Anchoring bias: staying with the first impression despite later information that contradicts it. Try to disprove the first diagnosis.
• Confirmation bias: looking for evidence that supports a diagnosis instead of testing alternatives.
• Framing bias: reaching a different conclusion from the same image depending on how the clinical history is presented. Look at the image first, then read the history.
• Availability bias: judging a diagnosis as likely because examples come to mind easily, such as a recent missed case.
• Attribution bias: attributing findings to a patient's characteristics, such as age or sex, given in the history.
• Premature closure: accepting a diagnosis as final too early in the evaluation.

Kim and Mansfield (2014) reviewed 656 examinations with a delayed diagnosis and classified 1,269 errors. The most common types were under-reading (42%) and satisfaction of search (22%). The missed diagnosis was also not recognised on later examinations in 196 of the 656 cases (30%).

Bias in imaging research

• Spectrum bias: a test is studied in a different range of patients from the people it is intended for, which changes its sensitivity and specificity.
• Partial reference (verification or work-up) bias: only some patients receive the reference standard, often those with a positive test.
• Differential reference bias: patients receive different reference standards.
• Incorporation bias: the result of the test being studied forms part of the reference standard.

Sensitivity, specificity and the design of diagnostic accuracy studies are covered in the diagnostic tests chapter.

Clinical trials

Which trial methods prevent which bias?

Each major bias in a randomised trial has a matching safeguard, and CONSORT 2025 asks authors to report it.

How easily would a significant result become non-significant?

The Fragility Index (Walsh and colleagues, 2014) is the number of patients in the group with fewer events whose status would have to change from non-event to event for the P-value to reach 0.05 or more. In 399 trials in high-impact journals the median was 8, and 25% had a Fragility Index of 3 or less.

Teaching example: 10 of 100 patients die with a new drug and 25 of 100 with control, Fisher's exact P = 0.0085. Changing one survivor at a time to a death in the drug group gives P = 0.016 at 11 deaths, 0.028 at 12, 0.046 at 13 and 0.073 at 14. The Fragility Index is 4.

By specialty

Bias problems by specialty

Common myths

Common mistaken beliefs

• ✗Myth: A large sample removes bias.

  ✓A larger sample reduces random error only. A biased estimate stays biased, with a narrower confidence interval.

• ✗Myth: Randomisation removes all bias.

  ✓Randomisation balances confounders at the start. Observer bias, loss to follow-up and selective reporting need blinding, complete follow-up and registration.

• ✗Myth: Adjusting for every available variable is the safest analysis.

  ✓Adjusting for a factor on the causal pathway removes part of the real effect. Choose adjustment variables before the analysis, using the three conditions for a confounder.

• ✗Myth: A reliable questionnaire is a valid questionnaire.

  ✓A tool can give the same wrong answer every time. Reliability is needed for validity but does not guarantee it.

• ✗Myth: Longer survival in screen-detected cases proves that screening works.

  ✓Lead-time and length-time bias lengthen survival after diagnosis without delaying death. Judge screening by deaths from the disease in a randomised trial.

• ✗Myth: A high correlation between two measurement methods shows that they agree.

  ✓Correlation is misleading for agreement (Bland and Altman, 1986). Two methods can be highly correlated and still give different values.

Why this matters to you

Five consequences of ignoring bias and confounding

1. Protocol sent back

   A case-control protocol with hospital controls and no plan for confounders is returned by the research committee.

2. A wrong thesis conclusion

   An unadjusted odds ratio produced by a confounder becomes the main finding of your thesis.

3. Questions you cannot answer at the viva

   Examiners ask how your controls were chosen, who measured the outcome, and which confounders you adjusted for.

4. Misreading papers

   Longer survival after screening and benefits seen in observational drug studies are often produced by bias or confounding.

5. Harm to patients

   Treatments adopted on the basis of biased studies may be ineffective or harmful.

History

History of bias, confounding and validity

1. 1946

   Joseph Berkson

   Showed that an association between diseases seen in hospital patients may not exist in the population, because of how patients come to be admitted.

2. 1951

   Lee Cronbach

   Published coefficient alpha and showed that it equals the mean of all possible split-half reliability coefficients of a test.

3. 1955

   Lee Cronbach and Paul Meehl

   Published "Construct validity in psychological tests", on how to judge a test that measures a concept with no single criterion to compare against.

4. 1957

   Donald Campbell

   Introduced the distinction between internal validity and external validity.

5. 1960

   Jacob Cohen

   Proposed kappa, a measure of agreement between two raters that allows for agreement expected by chance.

6. 1965

   Austin Bradford Hill

   Published "The environment and disease: association or causation?", with nine viewpoints for judging whether an association is causal.

7. 1976

   Anthony McMichael

   Described the "healthy worker effect": workers must be healthy enough to be employed, so comparing them with the general population underestimates their mortality.

8. 1978

   Roberts, Spitzer, Delmore and Sackett

   Published an empirical demonstration of Berkson's bias: respiratory and locomotor disease were associated in people who had been in hospital, but essentially independent in the community sample.

9. 1979

   David Sackett

   Listed 56 biases, grouped by the stage of research at which they arise, and proposed an annotated catalogue of bias.

10. 1986

    Martin Bland and Douglas Altman

    Showed that correlation is misleading when judging whether two measurement methods agree, and described an alternative graphical approach.

11. 1991

    Easterbrook and colleagues

    Found that studies with statistically significant results were more likely to be published (adjusted odds ratio 2.32).

12. 1997

    Egger and colleagues

    Proposed a simple test of funnel plot asymmetry to detect bias in meta-analyses.

13. 2008

    Samy Suissa

    Described immortal time bias in pharmaco-epidemiology: follow-up time during which, because of how exposure is defined, the outcome could not occur.

14. 2014

    Walsh and colleagues

    Proposed the Fragility Index for trials with a statistically significant binary or time-to-event outcome.

15. 2019

    Sterne and colleagues

    Published RoB 2, the revised Cochrane tool for assessing risk of bias in randomised trials.

Guidelines

Bias in guidelines

Reading a paper

Five checks for bias in any study

• How were participants selected, and were the groups comparable at the start?
• Were outcome assessors, and where possible participants and carers, blinded to the group?
• How many were lost to follow-up, and were the losses similar between groups?
• Which confounders were measured and adjusted for, and was any important one missing?
• Were the outcomes and analyses prespecified in a registered protocol?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

Every TNMGRMU question on bias, confounding and validity

Each one is answered in this chapter. Tap a sitting to open the university's own question paper.

1. ×1Confounding factors in clinical trialsMD Pharmacology · Oct 2025
2. ×1Types of Bias in vaccine trialsMD Pharmacology · Oct 2025
3. ×1Confounders vs Effect modifiersMD Pharmacology · Oct 2025
4. ×1Confounder and effect modifier in Epidemiological studiesMD Community Medicine · May 2025
5. ×1As a health researcher, you are planning to do health research among college students using questionnaire as the only data collecting tool. Discuss various components that you will consider while designing the questionnaire and the methods you will apply to test reliability and validity of a questionnaireMD Community Medicine · Oct 2025
6. ×1Biases in clinical researchDM Pulmonary Medicine · Aug 2025
7. ×1Biases in study designDM Clinical Immunology and Rheumatology · Feb 2025
8. ×1BIAS in clinical trialsMCh Gynaecological Oncology · Feb 2025
9. ×1Differentiate Effect Modifiers and Confounders in health research with suitable examplesMD Community Medicine · Oct 2024
10. ×1Reliability & ValidityPh.D. RM · Jul 2024
11. ×1Describe the steps in conducting Meta-Analysis and sources of bias in Meta - Analysis. How is Heterogenecity between studies addressed during analysis?MD Community Medicine · Oct 2023
12. ×1Confounding is not a bias –Justify the statementMD Community Medicine · Oct 2023
13. ×1Reliability and ValidityPh.D. RM · Jul 2023
14. ×1Biases in radiology reportingMD Radio Diagnosis · May 2022
15. ×1Effect Modifier vs ConfounderMD Community Medicine · Oct 2022
16. ×1Regarding randomised clinical trials, explain the following terms: a. External validity. b. Allocation concealment. c. Stratification. d. Sensitivity analysis. e. Fragility IndexDM Critical Care Medicine · Aug 2022
17. ×1Bias in ResearchPh.D. RM · Jul 2022
18. ×1Write briefly on how ‘selection bias’ can affect study inferences. List the methods used to minimize itMD Community Medicine · Jul 2021
19. ×1Describe the biases in cross sectional studiesMD Community Medicine · Jul 2021
20. ×1ConfoundingPh.D. RM · Aug 2021
21. ×1Describe the steps in the design of cohort studies. Discuss the biases that may affect cohort studiesMD Community Medicine · May 2019
22. ×1Elaborate on the concept of lead time bias in cancer screeningMCh Surgical Oncology · Feb 2019
23. ×1Bias in case control studiesMD Community Medicine · Oct 2016
24. ×1It is better to express imprecision of an analytical method in terms of coefficient of variation rather than standard deviation. Justify with an exampleMD Biochemistry · Oct 2014
25. ×1Confounding variablesMD Community Medicine · Oct 2014
26. ×1Define the term Validity. Discuss the different forms of ValidityPh.D. RM · Jul 2013
27. ×1Distinguish between “Accuracy” and “Precision”Ph.D. RM · Jul 2013
28. ×1Give the steps for an epidemiological study to find out cause / risk factors for a newly reported syndrome. What are the possible sources of Bias in this studyMD Community Medicine · Oct 2012
29. ×1Biases in medical researchMCh Surgical Oncology · Aug 2009
30. ×1Bias in studiesMD Community Medicine · Feb 2005
31. ×1How is a screening test for cancer evaluated? Discuss the biases caused as a result of a screening programmeMCh Surgical Oncology · Apr 2004

Further reading

Related Stateazy articles