How do you combine many studies?

Start here

Why not trust one trial, or one review article?

One trial can be too small to show an effect. A review article can cite only the studies that agree with its author. A systematic review finds all the studies on a question by a method written in advance, and a meta-analysis combines their numbers.

Comparison

Systematic review or narrative review: what is the difference?

Both summarise studies. They differ in how the studies are found and chosen.

Steps

How do you do a systematic review?

Each step is fixed in the protocol before you look at any results.

1. Frame the question with PICO: patients, intervention, comparison, outcome, and the study designs you will accept. For a question about an exposure rather than a treatment, the same frame is written as PECO: population, exposure, comparator, outcome (Morgan and colleagues, 2018).
2. Write the protocol and register it. PROSPERO, the international register run by the Centre for Reviews and Dissemination at York, is free. Its developers advised registering before formal screening starts.
3. Search several databases (for example PubMed and the Cochrane Library), trial registers such as CTRI and ClinicalTrials.gov, and reference lists. Record the full search strategy and the date of each search. See the literature search chapter.
4. Screen titles and abstracts, then full reports, against the eligibility criteria. Two reviewers work independently and settle disagreements.
5. Extract data on a piloted form: study design, patients, intervention, outcomes and results.
6. Assess risk of bias in every included study. Use RoB 2 for randomised trials and ROBINS-I for non-randomised studies of interventions.
7. Synthesise: a meta-analysis if the studies are similar enough, otherwise a structured description in tables.
8. Assess certainty of the evidence for each outcome, for example with GRADE, and look for missing studies.
9. Report with the PRISMA 2020 checklist and flow diagram, and update the review when new trials appear.

PRISMA 2020

How do you show which studies you kept?

The flow diagram accounts for every record, from the search to the final set of studies. PRISMA 2020 item 16a asks for this, ideally as a diagram.

Interactive

Change the counts and see which boxes no longer add up

Each box in the flow diagram follows from the boxes above it, so an error in one count breaks the arithmetic of the next box.

The formula

How are the results of several trials combined?

Not by adding the patients together, and not by counting how many trials were significant. Each trial's result is weighted by its precision, so a large trial counts for more than a small one.

weight w = 1 ÷ SE²

pooled log RR = Σ(w × log RR) ÷ Σw

RR

risk ratio in each trial: risk with drug ÷ risk with placebo

log RR

natural logarithm of the RR. Ratios are pooled on the log scale, then converted back

SE

standard error of log RR = √(1/a − 1/n₁ + 1/c − 1/n₂), where a of n₁ patients had the event on drug and c of n₂ on placebo

Σ

sum over all trials

This is the inverse-variance, fixed-effect method. Teaching example, Trial A in the forest plot below: 12 of 50 patients had PONV on the drug (0.24) and 20 of 50 on placebo (0.40).

RR = 0.24 ÷ 0.40 = 0.60 → log RR = −0.511

SE² = 1/12 − 1/50 + 1/20 − 1/50 = 0.0933 → w = 10.7

Repeat for all six trials, then combine:

Σw = 100.7, Σ(w × log RR) = −41.96 → pooled log RR = −0.417 → RR 0.66

SE = 1 ÷ √100.7 = 0.0997 → 95% CI = e^(−0.417 ± 1.96 × 0.0997) = 0.54 to 0.80

Other pooling methods exist, such as Mantel-Haenszel and Peto for yes/no outcomes. Software (RevMan, R) does the arithmetic, but you must be able to explain it.

Why not add up the patients from all the trials?

Adding the events and patients of every trial as if they came from one large trial can reverse the answer when the trials allocate patients unequally and have different baseline risks. Teaching example with made-up numbers.

Both trials show about 17% lower risk with the drug. Added together, the drug appears harmful, because most drug patients came from the high-risk trial and most control patients from the low-risk trial. A meta-analysis compares drug and control within each trial first, then pools the within-trial results. Altman and Deeks (2002) showed that treating the data as if they came from one trial is prone to this bias, Simpson's paradox, when groups are imbalanced within trials.

Reading the result

How do you read a forest plot?

One row per study, one diamond for the pooled result, and a vertical line where the drug has no effect.

Read a forest plot in this order: the outcome and effect measure, which side favours which treatment, the diamond and whether it crosses the line, the heterogeneity statistic (I²), and then the largest squares, which carry the most weight.

Heterogeneity

Do the studies agree with each other?

If the trials are measuring the same effect, their confidence intervals should overlap. When they do not, the difference between studies is more than chance, and one pooled number may hide it.

I² = 100% × (Q − df) ÷ Q

Q

Cochran’s Q: the weighted sum of squared differences between each study’s result and the pooled result

df

degrees of freedom = number of studies − 1

I²

the percentage of the variation between studies that is due to heterogeneity rather than chance. Negative values are set to 0%

Left panel: Q = 1.12 with df = 5. Q is smaller than df, so I² = 0%. Right panel: Q = 22.90, df = 5, so I² = 100% × (22.90 − 5) ÷ 22.90 = 78%. Higgins and colleagues (2003) tentatively called I² values of 25%, 50% and 75% low, moderate and high.

The Q test has low power when there are few studies, so a non-significant Q does not show that the studies agree. When heterogeneity is present, look for its cause: different patients, doses, follow-up or risk of bias. Explore it with subgroup analysis or meta-regression planned in the protocol, check with sensitivity analysis whether the result changes when some studies are left out, use a random-effects model, or do not pool.

Every extra subgroup analysis adds a chance of a false-positive finding. The Practical Statistics chapter on multiple comparisons explains how this risk grows with the number of tests and how to allow for it.

Comparison

Fixed effect or random effects: which model?

The two models make different assumptions about what the trials are estimating, and so they weight the trials differently.

Interactive

Remove trials or switch the model, and watch the pooled risk ratio and I² change

The same inverse-variance and DerSimonian-Laird calculations as above, repeated for the trials you keep.

Effect measures

What is pooled when trials measure the outcome in different ways?

Each trial's result must be expressed as the same effect measure before the results can be pooled.

The standardised mean difference (SMD) divides each trial's mean difference by the pooled SD in that trial. The result is in SD units, so trials that used different scales can be combined, provided the scales measure the same thing.

SMD = (mean with treatment − mean with control) ÷ pooled SD

Trial on a 0–10 pain scale: (3.2 − 4.1) ÷ 1.90 = −0.47

Trial on a 0–100 pain scale: (32 − 45) ÷ 21.0 = −0.62

Teaching examples with made-up numbers: SDs 1.8 and 2.0 with 40 patients per group, and SDs 20 and 22 with 50 per group. The mean differences of −0.9 and −13 points cannot be pooled directly, but −0.47 and −0.62 SD can. When a trial reports the median and interquartile range instead of the mean and SD, Wan and colleagues (2014) described how to estimate the mean and SD from these and the sample size.

Publication bias

Are some studies missing?

Trials with positive results are more likely to be published, published quickly, in English, more than once, and cited. If small negative trials stay unpublished, a meta-analysis of what is published overestimates the effect.

Asymmetry does not prove publication bias. Heterogeneity, lower methodological quality of small studies, and chance can also produce it. Egger’s regression test measures asymmetry, but tests for funnel plot asymmetry should not be used when a meta-analysis has fewer than 10 studies, because their power is too low (Sterne et al., 2011).

Rosenthal (1979) called this the file drawer problem. In its extreme form, journals publish the 5% of studies that found an effect by chance, while the studies with non-significant results stay in the researchers' file drawers.

Interactive

Hide small trials that found no benefit, and watch the pooled risk ratio move

The same 25 made-up trials as the funnel plot above.

Comparison

Published summary results, or the data of every patient?

Most meta-analyses pool the summary result of each trial. An individual patient data (IPD) meta-analysis obtains the raw data of every participant from the trial teams.

The analysis must keep each trial's patients together. Riley and colleagues (2010) stated that it is inappropriate to analyse individual participant data as if they all came from a single study. Either each trial is analysed first and the results are then pooled (two step), or all trials are modelled together while allowing for the trial each patient came from (one step).

Merits and limitations

What can a meta-analysis do, and what can it not do?

Quality and reporting

How do you judge a published systematic review?

Two different questions: was it reported completely, and was it done well? Different checklists answer each.

AMSTAR 2 critical domains. A critical flaw in any of these lowers confidence to low or critically low: protocol registered before the review (item 2), adequate literature search (item 4), justification for excluding individual studies (item 7), risk of bias of included studies (item 9), appropriate meta-analytical methods (item 11), risk of bias considered when interpreting the results (item 13), and assessment of publication bias (item 15).

To appraise a clinical practice guideline, use AGREE II (23 items and two overall assessments). General reporting guidelines are covered in the writing and referencing chapter.

Evidence-based medicine

Where do systematic reviews fit into patient care?

A systematic review supplies the research evidence. Evidence-based medicine is how you use it for one patient.

Sackett and colleagues (BMJ, 1996) defined it: “Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.” They added that good doctors use both clinical expertise and the best external evidence, “and neither alone is enough”.

The five steps (Sicily statement, 2005)

1. Ask: turn the uncertainty into an answerable question (PICO).
2. Acquire: search systematically for the best available evidence.
3. Appraise: judge its validity, clinical relevance and applicability.
4. Apply: use the results in practice with this patient.
5. Assess: evaluate your own performance.

Tools used at each step

Decision analysis

How do you weigh the chance of disease against what each outcome is worth?

A decision tree sets out each choice, the probability of each result and a value, called a utility, for each final outcome. The choice with the higher expected utility is preferred.

A utility is a number from 0 (death) to 1 (full health) that states how much an outcome is worth, ideally to the patient. The expected utility of a choice is the sum, over its outcomes, of each outcome's probability multiplied by its utility.

Disease, treated

utility 0.90

Disease, not treated

utility 0.50

No disease, treated (side effects only)

utility 0.95

No disease, not treated

utility 1.00

Treat: 0.3 × 0.90 + 0.7 × 0.95 = 0.935

Do not treat: 0.3 × 0.50 + 0.7 × 1.00 = 0.850

Treating gives the higher expected utility at a 30% probability of disease. The two choices are equal at the treatment threshold.

Threshold = harm ÷ (harm + benefit) = 0.05 ÷ (0.05 + 0.40) = 11.1%

benefit

the gain from treating a patient who has the disease, 0.90 − 0.50 = 0.40

harm

the loss from treating a patient who does not, 1.00 − 0.95 = 0.05

Below 11.1% the expected utility is higher without treatment, and above it with treatment. Repeating the calculation over a range of probabilities and utilities, as the right panel does, is a sensitivity analysis. It shows whether the choice changes within the plausible range. A test result that moves the probability across the threshold changes the decision, which is the idea behind the threshold approach in the diagnostic tests chapter.

Levels of evidence

Which kind of study should you believe more?

Designs that are less open to bias rank higher. But a well-done cohort study can give more certain evidence than a flawed trial, so modern schemes let the level move.

Oxford CEBM 2011 levels: “Does this intervention help?”

The 2011 table has separate rows for prevalence, diagnosis, prognosis, harms and screening. A level may be graded down for study quality, imprecision, indirectness, inconsistency or a very small effect, and graded up for a large effect.

GRADE (2008)

GRADE rates the quality (certainty) of evidence for each outcome as high, moderate, low or very low, and each recommendation as strong or weak. Evidence from randomised trials starts high and is rated down for study limitations, inconsistency, indirectness, imprecision and reporting bias. Observational evidence starts low and can be rated up for a very large effect, a dose-response relation, or when all plausible biases would reduce the apparent effect.

Decide

Which kind of review fits your question?

Answer a few questions. Each answer names the review type and its reporting guideline.

By specialty

Systematic review problems by specialty

Common myths

Common mistaken beliefs

• ✗Myth: A meta-analysis is always the highest level of evidence.

  ✓It is only as reliable as the studies in it. A meta-analysis of non-randomised studies does not carry the weight of one of well-conducted randomised trials.

• ✗Myth: Many small trials combined are as trustworthy as one large trial.

  ✓Small trials are more open to bias and to non-publication. Egger and colleagues (1997) found that when a meta-analysis disagreed with a later large trial, the meta-analysis had shown the larger effect every time.

• ✗Myth: Count the significant trials: 3 positive and 3 negative means the evidence is conflicting.

  ✓Vote counting ignores trial size. Six trials that are each too small can pool to a clear result.

• ✗Myth: A non-significant heterogeneity test means the studies agree.

  ✓The Q test has low power when there are few studies. Look at I² and at the forest plot.

• ✗Myth: An asymmetrical funnel plot proves publication bias.

  ✓Heterogeneity, poor quality of small studies and chance can also cause asymmetry.

• ✗Myth: A review that follows PRISMA is a good review.

  ✓PRISMA checks reporting, not conduct. Use AMSTAR 2 to judge quality.

Why this matters to you

Five consequences of not understanding systematic reviews

1. A biased review of literature

   A thesis review that cites only supportive studies is a narrative review, and the examiner will ask how the studies were chosen.

2. An unregistered review

   A systematic review thesis started without a registered protocol cannot show that its methods were fixed before the results were known.

3. Questions you cannot answer at the viva

   You cannot say what the diamond means, why a large trial has a larger square, or what I² = 78% implies.

4. Misreading papers

   You quote a pooled estimate without noticing that the trials disagree, or that small negative trials are missing.

5. Treatment that does not work

   Intravenous magnesium after myocardial infarction looked beneficial in a meta-analysis of small trials, and ISIS-4 found no benefit.

History

History of systematic reviews and meta-analysis

1. 1904

   Karl Pearson

   Published a report in the BMJ that combined results from several studies of typhoid (enteric fever) inoculation into a pooled estimate, without an estimate of its uncertainty.

2. 1976

   Gene Glass

   Coined the term "meta-analysis" for the statistical analysis of a large collection of results from individual studies, to integrate the findings.

3. 1979

   Robert Rosenthal

   Described the "file drawer problem": studies with non-significant results that stay unpublished.

4. 1986

   Rebecca DerSimonian and Nan Laird

   Described a random-effects approach that allows for differences in treatment effect between trials. It is still the simplest random-effects method.

5. 1992

   Evidence-Based Medicine Working Group

   Published "Evidence-based medicine. A new approach to teaching the practice of medicine" in JAMA.

6. 1993

   The Cochrane Collaboration

   Established in Oxford to prepare systematic reviews of randomised trials of health care. It is named after the Scottish researcher Archie Cochrane.

7. 1996

   David Sackett and colleagues

   Defined evidence-based medicine in the BMJ as the use of current best evidence in decisions about individual patients.

8. 1997

   Matthias Egger and colleagues

   Proposed a regression test for funnel plot asymmetry, and found asymmetry suggesting bias in 38% of meta-analyses from leading journals.

9. 1999–2000

   QUOROM and MOOSE

   Reporting standards for meta-analyses of randomised trials (QUOROM, 1999) and of observational studies (MOOSE, 2000).

10. 2002

    Julian Higgins and Simon Thompson

    Proposed I² to measure how much of the variation between studies is due to heterogeneity.

11. 2009

    PRISMA statement

    Replaced QUOROM as the reporting guideline for systematic reviews and meta-analyses.

12. 2011

    PROSPERO

    The international prospective register of systematic reviews was launched in February 2011 by the Centre for Reviews and Dissemination, University of York.

13. 2017

    AMSTAR 2

    Revised critical appraisal tool for systematic reviews that include randomised or non-randomised studies, with 16 items.

14. 2021

    PRISMA 2020

    Updated reporting guideline with a 27-item checklist and new flow diagrams, published in the BMJ by Page and colleagues.

Guidelines

Systematic reviews in guidelines

Reading a paper

Five checks on any systematic review or meta-analysis

• Was the protocol registered (PROSPERO or similar) before the search, and did the review follow it?
• Did the search cover several databases and trial registers, without language restrictions?
• Was risk of bias assessed for each study, and did it affect the conclusions?
• Do the trials agree? Look at the forest plot and I² before trusting the diamond.
• Were funnel plots or other checks used for missing studies, and were there at least 10 studies for a test?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

Every TNMGRMU question on systematic review and meta-analysis

Each one is answered in this chapter. Tap a sitting to open the university's own question paper.

1. ×18Meta analysisMCh Surgical Oncology · Feb 2023MD Pharmacology · Oct 2022MD Radio Therapy · Oct 2022MD Community Medicine · Oct 2022DM Medical Oncology · Feb 2022MCh Surgical Oncology · Feb 2022Ph.D. RM · Jul 2022MD Radio Therapy · Jul 2021MD Pharmacology · Oct 2019MCh Surgical Oncology · Aug 2019MD Pharmacology · May 2017MD Community Medicine · Apr 2016Ph.D. RM · Jan 2015MCh Surgical Oncology · Feb 2011MCh Surgical Oncology · Feb 2010MCh Surgical Oncology · Feb 2010DM Medical Oncology · Feb 2009DM Medical Oncology · Apr 1996
2. ×3Evidence Based MedicineMD Pharmacology · May 2025MD Pharmacology · Apr 2015DM Neurology · Aug 2009
3. ×3Forest plotDM Neonatology · Aug 2023DM Medical Oncology · Aug 2019DM Neonatology · Aug 2012
4. ×3Levels of evidenceMCh Surgical Oncology · Feb 2017MCh Urology · Feb 2014MCh Surgical Oncology · Feb 2012
5. ×2Write briefly on principles, strengths and weaknesses of meta-analysis, randomized controlled trials, cohort studies and case controlled studiesDM Clinical Immunology and Rheumatology · Feb 2023DM Clinical Immunology and Rheumatology · Aug 2021
6. ×1Draw a Pyramid of levels of evidence in evidence-based medicineMD Pharmacology · Oct 2025
7. ×1Discuss critical appraisal of a systematic review and guidelinesDM Pediatric Neurology · Feb 2025
8. ×1Discuss the reporting standard for systematic reviewMD Community Medicine · Oct 2024
9. ×1Define systematic review. Explain the steps of conducting systematic review. Compare and contrast Systematic and narrative reviewMD Community Medicine · Oct 2024
10. ×1What is systematic review and meta-analysis? Describe the process, merits and limitations of meta-analysisDM Pulmonary Medicine · Jan 2024
11. ×1Discuss Evidence Based Medicine and its strength and weaknessesDM Neonatology · Jan 2024
12. ×1Kaplan Meier curve and Funnel plotDM Neonatology · Jan 2024
13. ×1Interpretation of a Forest PlotDM Endocrinology · Aug 2024
14. ×1Describe the steps in conducting Meta-Analysis and sources of bias in Meta - Analysis. How is Heterogenecity between studies addressed during analysis?MD Community Medicine · Oct 2023
15. ×1Evidence based medicine is valued higher than Meta analysis –JustifyMD Community Medicine · Oct 2023
16. ×1Checklist for assessing quality of meta-analysis – discuss in briefMD Community Medicine · May 2023
17. ×1Define Evidence based medicine and their levels. Discuss the role of Outcome measures in Rehabilitations in relation to ICF DomainsMD Physical Medicine and Rehabilitation · May 2022
18. ×1Systematic Review vs narrator reviewMD Community Medicine · Oct 2022
19. ×1Describe Meta analysis and how to conduct and analyze the Meta analysisDM Neonatology · Feb 2022
20. ×1List the levels of evidence. What is meta-analysis? Explain meta-analysis with an exampleMD Radio Therapy · Nov 2020
21. ×1What is Evidence based medicine? Discuss various tools used in Evidence based medicineMD Community Medicine · Aug 2020
22. ×1What is a systematic review? Enumerate and discuss the steps required in writing a good systematic reviewDM Neonatology · Feb 2020
23. ×1How do you conduct Meta Analysis and explain it with an example?Ph.D. RM · Dec 2020
24. ×1Describe the steps involved in conducting a meta-analysisPh.D. RM · Jan 2019
25. ×1Practice of evidence based medicine modelMD Pharmacology · May 2018
26. ×1Levels of evidence and GRADE of recommendationDM Neonatology · Feb 2018
27. ×1Systemic reviewPh.D. RM · Jul 2016
28. ×1GWAS meta-analysis report in T2DMDM Nephrology · Feb 2015
29. ×1Evidence based medicine? Describe strength and weakness of evidence based medicineDM Neonatology · Aug 2013
30. ×1Meta-analysis and its uses in oncologyDM Medical Oncology · Aug 2013
31. ×1(a) Explain what is meant by ‘Multivariate Analysis’. (b) Explain in detail Partial Least Square Regression. (c) Explain what is Meta Analysis, the steps of Meta Analysis, the methods and assumptions of Meta AnalysisPh.D. RM · Jul 2013
32. ×1Evidence Based Medicine in Medical PracticeMD Community Medicine · Oct 2012
33. ×1Pitfalls in meta analysisMCh Surgical Oncology · Aug 2008
34. ×1Meta analysiaMCh Surgical Oncology · Apr 2004
35. ×1What is Evidence Based Medicine? How will you assess the strength of evidence for a treatment option in renal therapeutics? Describe the various treatment options and the quality of evidence of available therapy in primary FSGSin ChildrenDM Nephrology · Mar 2002
36. ×1Prion (4d) Meta analysisDM Neurology · Apr 1997
37. ×1What is meta analysis? Discuss its impact on current management of breast cancerDM Medical Oncology · Apr 1997
38. ×1How would you assess quality of life in cancer patients after therapy? (25) 1 3. Write short notes on: (a) Taxol. (b) Fludarabine. (c) Peripheral blood stem cell transplants. d) P53. (c) Meta Analysis. (05=01XS)DM Medical Oncology · Nov 1993
39. ×1Role of meta analysis in researchMD Community Medicine