Do two observers agree on categories?

Start here

You agree on 92% of films. Why is that not the whole answer?

Two readers agree on some films by chance alone. If most films are negative, two readers who call almost everything negative will agree most of the time without looking carefully.

Imagine two readers who do not look at the films at all. Each tosses a biased coin and calls 30% of films positive. Both call a film positive with probability 0.30 × 0.30 = 0.09 and both call it negative with probability 0.70 × 0.70 = 0.49. They agree on 58% of films with no skill at all. This is the agreement expected by chance, p_e. It is higher when one category is common.

Cohen (1960) proposed kappa to correct the observed agreement for chance. Kappa is 1 when agreement is perfect, 0 when agreement equals what chance would produce, and below 0 when the raters agree less often than chance. The basic 2 × 2 calculation was introduced in the bias and validity chapter. This chapter explains why kappa behaves as it does, and the methods that go further.

The calculation

How is Cohen's kappa calculated?

Three numbers: observed agreement, agreement expected by chance from each rater's totals, and kappa.

κ = (p_o − p_e) ÷ (1 − p_e)

p o

observed agreement: the sum of the diagonal cells ÷ n

p e

chance agreement: for each category, (row total ÷ n) × (column total ÷ n), added over the categories

1 − p e

the largest possible agreement beyond chance

1. Observed agreement: p_o = (20 + 65) ÷ 100 = 0.85.
2. Sonologist A reports stones in 25% of scans, sonologist B in 30%.
3. Chance agreement: p_e = 0.25 × 0.30 + 0.75 × 0.70 = 0.075 + 0.525 = 0.60. As counts: 7.5 + 52.5 = 60 of 100 scans.
4. κ = (0.85 − 0.60) ÷ (1 − 0.60) = 0.25 ÷ 0.40 = 0.625, which rounds to 0.63. 95% CI 0.45 to 0.80.

The same formula works for any number of categories: p_o is the sum of all diagonal cells and p_e adds row share × column share for every category. Unweighted kappa treats every disagreement as equally serious, so the order of the categories is ignored.

Interpretation

Is a kappa of 0.6 good?

Most papers quote the labels of Landis and Koch (1977). The authors themselves described the divisions as clearly arbitrary, offered as benchmarks.

• The labels are conventions. They were not derived from data about clinical consequences. A kappa of 0.6 may be acceptable for a symptom checklist and unacceptable for a biopsy diagnosis that decides surgery.
• Use the confidence interval. In the gallstone example κ = 0.63 with a 95% CI of 0.45 to 0.80: the interval covers three labels.
• Kappa depends on prevalence and on bias between raters (next section), so the same labels do not mean the same thing in two studies with different case mixes.
• Report the table. Observed agreement, the marginal totals and kappa together let a reader judge the result.

The kappa paradoxes

Why can 90% agreement give a low kappa?

Feinstein and Cicchetti (1990) described two paradoxes. In the first, a high observed agreement gives a low kappa when the marginal totals are strongly imbalanced, that is when one category is much more common. In the second, kappa is higher when the imbalance is asymmetrical between raters than when it is symmetrical.

Paradox 1: prevalence

When most films are positive (or most are negative), the chance agreement p_e is high. The room left for agreement beyond chance, 1 − p_e, is small, and the few disagreements take up much of it. In the problem at the top of this chapter, each radiologist called 7% of films positive: p_e = 0.07 × 0.07 + 0.93 × 0.93 = 0.87, so 92% observed agreement gives κ = 0.39.

Paradox 2: bias

Bias here means that one rater calls a category more often than the other. With the same observed agreement and the same average prevalence, a table with bias gives a slightly higher kappa: in the teaching tables, 750 · 50 · 50 · 150 gives κ = 0.69 and 750 · 100 · 0 · 150 gives κ = 0.69 as well (0.6875 against 0.6923), although in the second table all disagreements go one way.

What to report: bias index, prevalence index and PABAK

Byrt, Bishop and Carlin (1993) showed that kappa is affected by bias and prevalence as well as by agreement, and recommended reporting measures of bias and prevalence with kappa. For a 2 × 2 table with cells a, b, c, d:

prevalence index

(a − d) ÷ n: far from 0 when one category is common

bias index

(b − c) ÷ n: far from 0 when one rater calls positive more often

PABAK

prevalence-adjusted bias-adjusted kappa = 2p_o − 1. It depends only on observed agreement

In the radiograph problem: prevalence index (6 − 178) ÷ 200 = −0.86, bias index 0, PABAK = 2 × 0.92 − 1 = 0.84, kappa 0.39. PABAK is not a replacement for kappa: it describes what kappa would be with 50% prevalence and no bias. Cicchetti and Feinstein (1990) recommended that kappa should always be accompanied by the separate proportions of agreement on positive and on negative decisions (p_pos and p_neg).

Interactive

Keep agreement at 90%, raise the prevalence and watch kappa fall

Observed agreement, prevalence and bias can be set separately. Kappa responds to all three. PABAK responds only to observed agreement.

Ordered categories

What if the grades are ordered?

When pathologists grade dysplasia as none, low grade or high grade, calling a high-grade lesion low grade is a smaller error than calling it normal. Unweighted kappa counts both as simple disagreement. Weighted kappa (Cohen, 1968) gives partial credit to near misses.

κ_w = (weighted p_o − weighted p_e) ÷ (1 − weighted p_e)

Each cell's share of patients (observed) and its chance share (row share × column share) are multiplied by its weight and added. With weights of 1 on the diagonal and 0 elsewhere, this is ordinary kappa.

• The weights change the answer. The same table gives 0.66, 0.72 or 0.78. Decide the weights in the protocol and state them in the paper.
• Quadratic weights and the ICC. Fleiss and Cohen (1973) showed that weighted kappa with quadratic weights is equivalent to an intraclass correlation coefficient, which links this chapter to the ICC chapter.
• Software defaults differ. SPSS Weighted Kappa uses linear weights by default. In R, irr::kappa2 gives unweighted kappa unless told otherwise.

Interactive

Move the disagreements one grade or two grades apart and compare the three kappas

Observed agreement on the diagonal stays at 78 of 100 in all three presets. Only the distance of the disagreements changes.

More than two raters

What if more than two raters classify each patient?

Fleiss (1971) extended kappa to many raters, with the same number of ratings for every patient. The raters rating one patient need not be the same people who rate the next.

For each patient, count how many of the m raters chose each category. Then:

P i

agreement for patient i = (sum of the squared counts − m) ÷ [m(m − 1)]: the share of rater pairs that agree

P̄

mean of P i over all patients: observed agreement

p j

share of all ratings that fell in category j

P̄ e

sum of p j²: chance agreement

Fleiss' κ = (P̄ − P̄_e) ÷ (1 − P̄_e)

Teaching example (made up): 4 cardiologists classify 10 ECGs as normal, ischaemic or other abnormality. For an ECG called normal by 3 and ischaemic by 1: P_i = (3² + 1² − 4) ÷ (4 × 3) = 0.50. Across the 10 ECGs P̄ = 0.70. The categories received 35%, 40% and 25% of the 40 ratings, so P̄_e = 0.35² + 0.40² + 0.25² = 0.345.

κ = (0.70 − 0.345) ÷ (1 − 0.345) = 0.355 ÷ 0.655 = 0.54

In the SPSS implementation no weights can be assigned to the disagreements, so Fleiss' kappa treats categories as nominal. The same prevalence effects apply as for Cohen's kappa.

Paired proportions

Did the two tests find the same proportion positive?

When the same patients have two tests, or the same test before and after treatment, the results are paired. McNemar (1947) described the test for the difference between such correlated proportions.

McNemar χ² = (b − c)² ÷ (b + c), with 1 degree of freedom

b

A positive, B negative (25)

c

A negative, B positive (10)

with continuity correction

(|b − c| − 1)² ÷ (b + c)

exact test

if b + c is small: the binomial probability of a split at least as uneven as b : c when each discordant pair is equally likely to go either way

1. Test A positive: (60 + 25) ÷ 200 = 42.5%. Test B positive: (60 + 10) ÷ 200 = 35.0%. Difference 7.5 percentage points = (b − c) ÷ n.
2. χ² = (25 − 10)² ÷ 35 = 225 ÷ 35 = 6.43, p = 0.011.
3. With continuity correction: (15 − 1)² ÷ 35 = 5.60, p = 0.018. Exact binomial test: p = 0.017.
4. Test A finds more positives than test B. On the same table, kappa = 0.63 (95% CI 0.52 to 0.74): the tests also agree moderately well on individual patients.

Why not the ordinary chi-square test? The ordinary chi-square test assumes two independent groups. Here the 200 results under test A and the 200 under test B come from the same 200 patients. The pairing is used by counting pairs, not individual results. See also the choosing a test chapter.

Interactive

Edit a paired table and see kappa and the McNemar test give different answers

Kappa asks whether the tests agree on each patient. McNemar asks whether they find the same proportion positive. A table can pass one and fail the other.

Measures of association

How strongly are two categorical variables related?

Agreement compares two ratings of the same thing. Association asks whether two different variables vary together, such as smoking status and COPD. The chi-square test shows whether an association exists. Its size depends on the sample size, so separate measures describe the strength.

χ² = 28.11 with (2 − 1) × (3 − 1) = 2 degrees of freedom, p < 0.001. All expected counts are 16.8 or more.

None of these is a measure of agreement. Two raters who always give opposite answers produce a strong association and no agreement. For a 2 × 2 table of two ratings, phi and kappa can be close in value (0.63 for both in the gallstone example) but they answer different questions. The odds ratio and relative risk, the other measures of association for 2 × 2 tables, are covered in the risk and survival chapter.

Decide

Which statistic do you need?

Answer the questions about your data.

Comparison

Pairs of methods that residents confuse

The formula

Worked examples, step by step

Made-up teaching data. Every number was computed in Python with statsmodels 0.14, SciPy 1.15 and scikit-learn.

Example 1. Cohen's kappa with its companions (gallstones, 100 scans)

1. Cells: a = 20 (both stones), b = 5 (A only), c = 10 (B only), d = 65 (neither).
2. p_o = 85 ÷ 100 = 0.85. p_e = (25 × 30 + 75 × 70) ÷ 100² = 6,000 ÷ 10,000 = 0.60.
3. κ = 0.25 ÷ 0.40 = 0.63. Standard error 0.087, 95% CI 0.45 to 0.80.
4. Prevalence index = (20 − 65) ÷ 100 = −0.45. Bias index = (5 − 10) ÷ 100 = −0.05. PABAK = 2 × 0.85 − 1 = 0.70.

κ = (0.85 − 0.60) ÷ (1 − 0.60) = 0.63 · PABAK = 0.70

Example 2. The radiograph problem (200 films)

1. Cells: a = 6, b = 8, c = 8, d = 178. p_o = 184 ÷ 200 = 0.92.
2. Each radiologist calls 14 ÷ 200 = 0.07 positive. p_e = 0.07² + 0.93² = 0.0049 + 0.8649 = 0.870.
3. κ = (0.92 − 0.870) ÷ (1 − 0.870) = 0.050 ÷ 0.130 = 0.39, 95% CI 0.14 to 0.63.
4. Prevalence index −0.86, bias index 0, PABAK 0.84. The low kappa reflects the rarity of positive films as well as the 16 disagreements, 8 in each direction.

Example 3. Weighted kappa (dysplasia, 100 biopsies)

Diagonal 40 + 20 + 18 = 78. Disagreements one grade apart: 6 + 5 + 4 + 5 = 20. Two grades apart: 1 + 1 = 2. Unweighted κ = (0.78 − 0.361) ÷ (1 − 0.361) = 0.66. Linear κ_w = (0.88 − 0.571) ÷ (1 − 0.571) = 0.72. Quadratic κ_w = (0.93 − 0.676) ÷ (1 − 0.676) = 0.78.

Example 4. McNemar test (two tests, 200 patients)

χ² = (25 − 10)² ÷ (25 + 10) = 6.43, df 1, p = 0.011 · corrected χ² = 5.60, p = 0.018 · exact p = 0.017

Example 5. Cramér's V (smoking and COPD, 250 adults)

V = √[28.11 ÷ (250 × 1)] = 0.34 · C = √[28.11 ÷ (28.11 + 250)] = 0.32

In a real paper

Kappa and the McNemar test in published studies

How to run it

Running kappa, weighted kappa, Fleiss' kappa and the McNemar test

Menu paths and function arguments below were checked against the IBM SPSS Statistics 30 and 31 documentation, the CRAN reference manuals and the jamovi documentation.

SPSS

1. Cohen's kappa and McNemar: Analyze > Descriptive Statistics > Crosstabs... Put rater A (or test A) in Rows and rater B in Columns. Click Statistics and tick Kappa and/or McNemar. Kappa needs a square table with the same categories in rows and columns, and both variables must have the same storage type (string or numeric). For tables larger than 2 × 2, the McNemar option reports the McNemar-Bowker test of symmetry.
2. Phi, Cramér's V and the contingency coefficient: in the same Statistics dialog, under Nominal.
3. Weighted kappa: Analyze > Scale > Weighted Kappa... Select the rater variables. Click Criteria to choose the weighting scale: linear (Cicchetti-Allison, the default) or quadratic.
4. Fleiss' kappa: Analyze > Scale > Reliability Analysis... Click Statistics, and under Interrater Agreement: Fleiss' Kappa select at least two rating variables. No weights can be assigned.

R

# ratings: one row per patient, one column per rater
library(irr)
kappa2(ratings)                          # Cohen's kappa, unweighted by default
kappa2(ratings, weight = "squared")      # weighted by squared distance (quadratic)
kappam.fleiss(ratings)                   # Fleiss' kappa for three or more raters

library(DescTools)
tab <- table(ratings$A, ratings$B)       # square table with the same levels
CohenKappa(tab, weights = "Equal-Spacing", conf.level = 0.95)   # linear weights
CohenKappa(tab, weights = "Fleiss-Cohen", conf.level = 0.95)    # quadratic weights

library(epiR)
epi.kappa(tab2x2, method = "fleiss")     # 2 x 2 table: kappa and PABAK

mcnemar.test(tab2x2)                     # continuity correction applied (correct = TRUE)
mcnemar.test(tab2x2, correct = FALSE)

library(vcd)
assocstats(tab)                          # phi, contingency coefficient, Cramér's V

jamovi

• McNemar test: the Paired Samples Contingency Tables analysis (jmv function contTablesPaired).
• Phi, Cramér's V and the contingency coefficient: options in the Contingency Tables analysis.
• Cohen's, weighted and Fleiss' kappa: the meddecide module (part of ClinicoPath) from the jamovi library.

How to report it

What to write in the thesis or paper

A kappa on its own cannot be interpreted. Give the reader the table, or the numbers needed to rebuild it.

1. Who rated, how they were trained, whether they rated independently and blind to each other and to clinical information.
2. The number of patients and raters, and how patients were selected (this sets the prevalence).
3. The agreement table, or observed agreement and each rater's proportion in each category.
4. Kappa with its 95% CI. For ordered categories, weighted kappa and the weights used. For more than two raters, Fleiss' kappa.
5. For rare or very common findings, the prevalence and bias indices and PABAK beside kappa.
6. For paired proportions, both proportions, their difference with a CI, and the McNemar test (state whether exact or with continuity correction).

Worked sentences, using the teaching examples:

“Two sonologists, blinded to each other's reports, independently assessed 100 scans for gallstones. They agreed on 85 scans (85%), with 60% agreement expected by chance. Cohen's kappa was 0.63 (95% CI 0.45 to 0.80). Sonologist A reported stones in 25% of scans and sonologist B in 30% (prevalence index −0.45, bias index −0.05, PABAK 0.70).”

“In the same 200 patients, test A was positive in 85 (42.5%) and test B in 70 (35.0%), a difference of 7.5 percentage points (exact McNemar test, p = 0.017).”

Reporting items for agreement studies (GRRAS) are listed in the guidelines section.

By specialty

Agreement on categories by specialty

Common myths

Common mistaken beliefs

• ✗Myth: 92% agreement shows that the observers agree well.

  ✓Part of that agreement is expected by chance. With a rare finding, chance agreement can be close to 90%.

• ✗Myth: A kappa of 0.41 is moderate, by rule.

  ✓The Landis and Koch labels are conventions that the authors called clearly arbitrary. Judge kappa against the clinical use and its confidence interval.

• ✗Myth: A low kappa with high agreement means the observers are careless.

  ✓It can come from extreme prevalence (the first kappa paradox). Report observed agreement, the prevalence and bias indices and PABAK with kappa.

• ✗Myth: The McNemar test measures agreement.

  ✓It tests whether two paired proportions differ. A table with no agreement beyond chance (κ = 0) can give McNemar p = 1.

• ✗Myth: A significant chi-square test on the rating table shows agreement.

  ✓Chi-square tests association. Raters who always disagree in a consistent way produce a strong association.

• ✗Myth: Unweighted kappa is fine for ordered grades.

  ✓It treats a one-grade and a two-grade disagreement as equally wrong. Use weighted kappa and state the weights.

Why this matters to you

Five consequences of misusing kappa

1. Protocol sent back

   A reliability objective without a plan for kappa, its CI, the weights (for grades) and the expected prevalence cannot be judged by the committee.

2. A wrong thesis conclusion

   Reporting percentage agreement alone makes rare findings look reliable. Reporting kappa alone makes them look unreliable.

3. Questions you cannot answer at the viva

   "Why is your kappa low when you agreed on 92%?" needs an answer in terms of chance agreement and prevalence.

4. Misreading papers

   Comparing kappas from studies with different case mixes, or treating a significant McNemar test as poor agreement.

5. Harm to patients

   A grading system with poor agreement beyond chance can send similar patients to different treatments.

History

History of agreement statistics for categories

1. 1947

   Quinn McNemar

   Published "Note on the sampling error of the difference between correlated proportions or percentages" in Psychometrika.

2. 1960

   Jacob Cohen

   Proposed "A coefficient of agreement for nominal scales": kappa, agreement corrected for chance.

3. 1968

   Jacob Cohen

   Described weighted kappa, with provision for scaled disagreement or partial credit.

4. 1971

   Joseph Fleiss

   Extended kappa to agreement among many raters.

5. 1973

   Joseph Fleiss and Jacob Cohen

   Showed the equivalence of weighted kappa (with quadratic weights) and the intraclass correlation coefficient as measures of reliability.

6. 1977

   J. Richard Landis and Gary Koch

   Published methods for observer agreement on categorical data and the benchmark labels from slight to almost perfect, describing the divisions as clearly arbitrary.

7. 1990

   Alvan Feinstein and Domenic Cicchetti

   Described the two paradoxes of high agreement but low kappa, and recommended reporting agreement on positive and negative decisions separately.

8. 1993

   Ted Byrt, Janet Bishop and John Carlin

   Defined indices of bias and prevalence, expressed kappa in terms of them, and recommended reporting them with kappa.

9. 2005

   Julius Sim and Chris Wright

   Reviewed kappa in reliability studies, including the effects of prevalence, bias and non-independent ratings, and sample sizes.

10. 2011

    Jan Kottner and colleagues

    Proposed the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).

Guidelines

Reporting agreement on categories

Reading a paper

Five checks on any kappa or McNemar result

• Is the agreement table, or observed agreement and the marginal totals, reported with kappa?
• Is kappa given with a 95% CI, and is the prevalence of each category shown?
• For ordered grades, was weighted kappa used, and are the weights named?
• Did the raters work independently and blind to each other and to the reference result?
• For paired proportions, was a paired test (McNemar) used, and was it read as a comparison of proportions, not as agreement?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

TNMGRMU questions on kappa and agreement on categories

Questions from MD, MS, DM, MCh and Ph.D. papers that this chapter helps you answer. Tap a sitting to open the university's own question paper.

1. ×1McNemar’s TestMD Community Medicine · Oct 2015