AI for Doctors / Theory / Lesson 5

ROC curves, AUC and the cut-off nobody tells you about

Why a great AUC can still mean a poor tool, and what calibration adds.

10 min

Most AI tools do not say yes or no. They give a score, and someone chooses a cut-off above which the score counts as positive. Every cut-off gives one pair of sensitivity and specificity. Plot all the pairs and you get the ROC curve; the area under it (AUC) summarises how well the scores separate the sick from the well.

Move the cut-off
no diseasedisease
1 − specificitysensitivity
Sensitivity79%
Specificity79%
AUC0.87

The AUC belongs to the model; the cut-off is a choice. Sliding it trades missed cases for false alarms, and the AUC stays the same. When a paper quotes only the AUC, ask which cut-off the hospital will actually use.

An AUC of 0.5 is a coin toss; 1.0 is perfect separation. But AUC says nothing about which cut-off the tool will run at, and nothing about whether a score of 0.8 really means an 80% risk. That second property is calibration. A well-calibrated model that says 20% risk is right about 20% of the time. A badly calibrated one can rank patients well (high AUC) and still give misleading risk numbers.

When the disease is rare, the precision-recall curve (PPV against sensitivity) is often more honest than the ROC curve, because it shows the false-alarm problem directly.

Decision curve analysis: a way of asking whether using the model at a given threshold does more good than treating everyone or no one. Increasingly expected in prediction-model papers.

Watch

ROC curve vs precision-recall curve
4 min · on-screen captions, no voice

Check yourself

1. Moving the cut-off changes:
2. A model has AUC 0.90 but says "30% risk" for a group in which 60% have the outcome. Its problem is:
3. For a rare disease, which curve better shows the false-alarm burden?

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.