Reading an AI paper in 15 minutes

The reporting guidelines journals now expect, and twelve questions that sort strong evidence from hype.

10 min

Clinical AI papers now have their own reporting guidelines, extensions of the ones you already know. If a paper does not mention the relevant one, that is itself a warning.

TRIPOD+AI (2024): prediction models built with regression or machine learning.
CONSORT-AI and SPIRIT-AI (2020): randomised trials of AI interventions and their protocols.
STARD-AI: diagnostic accuracy studies of AI.
CLAIM: AI in medical imaging (updated 2024).
DECIDE-AI (2022): early live clinical evaluation of AI decision support.
PROBAST+AI: risk of bias in prediction-model studies.
TRIPOD-LLM: studies that use large language models.

The ladder of evidence matters more than the AUC. Internal validation is the weakest rung, then external validation in another hospital, then prospective use, then a randomised trial with patient outcomes. Most published clinical AI never gets past the second rung.

Is the clinical question and intended user clear (who acts on the output, and when)?Were patients and data sources described, including where and when the data came from?Was the reference standard (ground truth) sound and applied to everyone?Were training, tuning and test data kept fully separate, at patient level?Was there external validation in a different hospital or time period?Are sensitivity, specificity, PPV and calibration reported, not just AUC?Is the cut-off pre-specified and justified?Is performance shown for subgroups (sex, age, skin tone, device, site)?Was the AI compared with clinicians, or clinicians with and without AI?Is there evidence on patient outcomes, not only accuracy?Are code, model or data available, or is access explained?Does the paper follow a reporting guideline (TRIPOD+AI, CONSORT-AI, STARD-AI, CLAIM)?

Yes0 / 12

Weak evidence. Treat the headline numbers with suspicion.

Check yourself

1. Which guideline covers an RCT of an AI intervention?

2. Strongest evidence for a clinical AI tool:

3. A paper reports only AUC. You should ask for:

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.