Do two things move together?

Start here

Do two measurements move together?

Plot one measurement against the other, one dot per patient. The correlation coefficient, r, describes the pattern with one number between −1 and +1.

r = 0 means there is no straight-line relation. It does not rule out a curved one. For x from −5 to 5 and y = x², r is exactly 0, although y depends completely on x. Draw the scatter plot before you calculate r.

Two types of correlation

Pearson or Spearman: which coefficient do you need?

Pearson's r measures how close the points are to a straight line. Spearman's ρ (rho) first ranks each variable and then correlates the ranks, so it measures whether one variable consistently rises or falls as the other rises.

r = Σ(x − x)(y − y) ÷ √[Σ(x − x)² × Σ(y − y)²]

ρ = 1 − 6Σd² ÷ [n(n² − 1)]

x, y

the means of x and y

Σ(x − x̄)(y − ȳ)

the sum of the products of the deviations from the means, the "product-moment" in the full name

d

the difference between each patient's rank on x and rank on y (the short formula is exact when there are no tied ranks)

n

the number of pairs

The numerator of r divided by n − 1 is the covariance, cov(x, y) = Σ(x − x)(y − y) ÷ (n − 1). Its sign gives the direction of the relation, but its size depends on the units: recording age in months instead of years multiplies it by 12. Dividing by the two SDs removes the units, so r = cov(x, y) ÷ (SD of x × SD of y). In the worked example below, cov = 1,200 ÷ 9 = 133.3 mmHg-years, the SD of age is 15.14 years and the SD of SBP is 10.69 mmHg, so r = 133.3 ÷ (15.14 × 10.69) = 0.82.

A third rank coefficient, Kendall's τ (tau), compares every pair of patients and counts how many are in the same order on both variables and how many are in the opposite order.

Interactive

Add or remove points and watch Pearson's r and Spearman's ρ change

Interpreting r

What do r and r² tell you?

r gives the direction and closeness of a straight-line relation. r², the coefficient of determination, is the proportion of the variation in one variable that the straight line accounts for.

Is r statistically significant?

t = r × √(n − 2) ÷ √(1 − r²), with n − 2 degrees of freedom

The test asks whether r could be 0 in the population. It says nothing about strength. With r = 0.10, the p value is 0.49 in 50 patients and 0.0015 in 1,000 patients. In both cases the line accounts for 1% of the variation. In the opening case, r = −0.32 in 200 patients gives t = −4.75 and p < 0.001, which shows that r is unlikely to be 0, not that the relation is strong.

Report r with its 95% confidence interval and n. The SAMPL guidelines advise against calling a correlation low, moderate or high unless the ranges for those words have been defined.

When r misleads

• The relation is curved: r underestimates it.
• One or two outliers: r can be raised or lowered by a single patient.
• A narrow range of values, for example only patients with normal blood pressure: r is lower than in the full population.
• Repeated measurements from the same patients counted as separate pairs: the pairs are not independent.

Interactive

Draw samples and watch how r and its p value vary with sample size

Correlation and cause

If two things correlate, does one cause the other?

Not necessarily. Before you call a correlation a cause, rule out chance, reverse causation, bias and confounding.

• Chance: test enough pairs of variables and some will correlate by chance alone.
• Reverse causation: the outcome can change the exposure. Patients who are unwell go outdoors less and have lower vitamin D.
• Confounding: a third factor affects both variables, as smoking does in the figure.
• Selection bias: the way patients were chosen can create a correlation that does not exist in the population.
• Regression toward the mean: patients chosen because a first reading was extreme will, on average, read closer to the mean the next time, even without treatment. A fall in blood pressure among patients recruited for a high first reading is partly this effect, which is one reason a control group is needed. It is the effect Galton described in 1886 (see History). It is covered with other misleading patterns in the Practical Statistics chapter on statistical fallacies.

A partial correlation is the correlation between two variables with a third held constant. For one variable z, r_xy·z = (r_xy − r_xz × r_yz) ÷ √[(1 − r_xz²)(1 − r_yz²)]. In the figure, holding smoking constant by looking within smokers and within non-smokers removes the correlation between the two scores. Adjusting for several variables at once is done with multiple regression, covered in the Practical Statistics chapter on multiple regression.

Designs that deal with these explanations are covered in the chapters on study designs and bias and validity. A randomised trial of vitamin D supplements, not a correlation, would show whether raising vitamin D lowers HbA1c.

Comparison

Correlation or regression: what is the difference?

Correlation describes how closely two variables vary together. Regression describes how much the outcome changes for each unit change in a predictor, and gives an equation to predict it.

The formula

How is the regression line calculated?

Simple linear regression fits the straight line that makes the sum of the squared vertical distances from the points to the line as small as possible. This is the method of least squares.

y = a + b × x

b = Σ(x − x)(y − y) ÷ Σ(x − x)²    a = y − b × x

y

the outcome (dependent variable)

x

the predictor (independent variable)

b

the regression coefficient, or slope: the change in y for a 1-unit rise in x

a

the intercept: the predicted y when x = 0

residual

observed y minus the y predicted by the line

Worked example (teaching data)

mean age = 52.5   mean SBP = 134.2

Σ(x − x)(y − y) = 1,200.0   Σ(x − x)² = 2,062.5   Σ(y − y)² = 1,027.6

b = 1,200.0 ÷ 2,062.5 = 0.58 mmHg per year

a = 134.2 − 0.5818 × 52.5 = 103.65 mmHg

r = 1,200.0 ÷ √(2,062.5 × 1,027.6) = 0.82 → r² = 0.68

b = 0.58 mmHg per year (95% CI 0.26 to 0.91, p = 0.003). The confidence interval does not include 0. The same calculation in Python:

from scipy import stats
age = [30, 35, 40, 45, 50, 55, 60, 65, 70, 75]
sbp = [118, 130, 121, 135, 128, 142, 131, 150, 139, 148]
fit = stats.linregress(age, sbp)
print(fit.slope, fit.intercept, fit.rvalue, fit.pvalue)
# slope 0.582, intercept 103.65, r 0.824, p 0.003

How well does the line fit?

SST = Σ(y − y)² = 1,027.6 (all the variation in SBP)

SSE = Σ(y − predicted y)² = 329.4 (left in the residuals)

SSR = SST − SSE = 698.2 (accounted for by the line)

R² = SSR ÷ SST = 698.2 ÷ 1,027.6 = 0.68, the same as r²

The residual SD, also called the standard error of the estimate, is s = √[SSE ÷ (n − 2)] = √(329.4 ÷ 8) = 6.42 mmHg. Observed SBP lies typically about 6 mmHg from the line, compared with an SD of 10.69 mmHg around the mean SBP.

SE of b = s ÷ √Σ(x − x)² = 6.42 ÷ √2,062.5 = 0.141

t = b ÷ SE of b = 0.582 ÷ 0.141 = 4.12, df 8, p = 0.003

This t is the same as the t for r (4.12 with 8 df), because in simple linear regression the test of b = 0 and the test of r = 0 are the same test. The 95% CI for b is 0.582 ± 2.306 × 0.141 = 0.26 to 0.91.

Assumptions to check

• The relation is a straight line (look at the scatter plot).
• Observations are independent: one row per patient.
• The residuals have a similar spread at all values of x (plot residuals against predicted values).
• The residuals are roughly normal, for confidence intervals and p values.
• Predict only within the range studied. This line describes ages 30 to 75.

Interactive

Move the line and watch the sum of squared residuals

Checking the model

How do residuals show whether a straight line is the right model?

A residual is the observed value minus the value predicted by the line. Plot the residuals against the predicted values. If the model is right, they form a band of even width around 0, with no pattern.

A high r or R² does not show that the model is right. The curved data in the middle panel have r = 0.70, yet a straight line is the wrong shape for them. The SAMPL guidelines ask authors to state whether an analysis of residuals confirmed the assumptions of linearity (see Guidelines).

Prediction

How precise is a prediction, and why not predict beyond the data?

A regression line gives two different intervals. The confidence interval shows where the mean outcome lies for patients with a given x. The prediction interval shows where one new patient's value is likely to lie, and it is always wider.

95% CI for the mean at x₀ = ŷ ± t × s × √[1/n + (x₀ − x)² ÷ Sxx]

95% prediction interval = ŷ ± t × s × √[1 + 1/n + (x₀ − x)² ÷ Sxx]

ŷ

the value predicted by the line at x₀

t

from the t distribution with n − 2 df: 2.306 for 8 df

s

the residual SD, 6.42 mmHg in the worked example

Sxx

Σ(x − x)², 2,062.5 in the worked example

Worked example at age 60: ŷ = 138.6 mmHg. The 95% CI for the mean SBP of 60-year-olds is 133.3 to 143.8 mmHg. The 95% prediction interval for one 60-year-old is 122.9 to 154.3 mmHg. Because of the term (x₀ − x)², both intervals widen as x₀ moves away from the mean age of 52.5.

Extrapolation. The ten adults were 30 to 75 years old. At 90 years the formula gives 156.0 mmHg (95% CI 142.9 to 169.1), but that calculation assumes the straight line continues beyond 75, and the data cannot show whether it does. Predict only within the range of x that was studied.

The intercept. a = 103.65 mmHg is the predicted SBP at age 0. No one near age 0 was studied, so here the intercept only fixes the position of the line and has no clinical meaning.

Interactive

Move the age and watch the confidence and prediction intervals widen

Both intervals are narrowest at the mean age. Beyond the ages studied they keep widening, and they also rest on an assumption the data cannot check.

More than one predictor

What if several factors affect the outcome?

Multiple linear regression puts several predictors in one equation. Each coefficient is then the change in the outcome for a 1-unit change in that predictor, with the other predictors held constant.

y = a + b₁x₁ + b₂x₂ + … + bₖxₖ

SBP = 57.8 + 0.52 × age + 1.90 × BMI   (R² = 0.94)

Teaching example: the same 10 adults with BMI added, fitted in Python (statsmodels). Age alone gave 0.58 mmHg per year. Adjusted for BMI, the coefficient for age is 0.52 mmHg per year (95% CI 0.37 to 0.68), and each extra kg/m² of BMI adds 1.90 mmHg (95% CI 1.10 to 2.71). A 60-year-old with a BMI of 25 has a predicted SBP of 136.8 mmHg.

• Uses: adjusting for confounders, predicting an outcome from several factors, and finding which factors are associated with the outcome independently of the others.
• Multicollinearity: when predictors are strongly correlated with each other, their coefficients become unstable. How to detect it and what to do is in the Practical Statistics chapter on multicollinearity.
• Overfitting: too many predictors for the number of patients fit chance patterns that do not repeat in new patients.
• R² always rises when a predictor is added. Adjusted R² (0.92 here) allows for the number of predictors.
• Multiple R is the correlation between the observed outcome and the value predicted by the model. Its square is R². With one predictor, R equals the size of r without its sign.
• Choose predictors from clinical knowledge before the analysis, and report how they were selected. Building and checking models with several predictors is covered in the Practical Statistics chapter on multiple regression.

Yes-or-no outcomes

What if the outcome is yes or no?

A straight line can predict probabilities below 0 or above 1. Logistic regression models the log of the odds instead, so the predicted probability follows an S-shaped curve that stays between 0 and 1.

ln[p ÷ (1 − p)] = a + b₁x₁ + b₂x₂ + … + bₖxₖ

odds ratio for a 1-unit rise in x = e^b   95% CI = e^{(b ± 1.96 × SE)}

p

probability of the outcome

p ÷ (1 − p)

the odds of the outcome

ln

natural logarithm: ln(odds) is the logit

e^b

the odds ratio for that predictor, adjusted for the others in the model

Worked example (teaching data)

Wound infection after surgery: 20 of 100 patients with diabetes and 20 of 200 patients without diabetes.

odds with diabetes = 20 ÷ 80 = 0.25   odds without = 20 ÷ 180 = 0.111

OR = 0.25 ÷ 0.111 = 2.25 → b = ln 2.25 = 0.811

A logistic regression with diabetes (1 = yes, 0 = no) as the only predictor gives the same result: intercept −2.197, coefficient 0.811, OR 2.25 (95% CI 1.15 to 4.41). If age and BMI are added to the model, the OR for diabetes becomes an adjusted odds ratio, which compares patients of the same age and BMI.

How many predictors can the model take?

Peduzzi and colleagues (1996) simulated logistic models and found that with fewer than 10 events per predictor variable the coefficients were biased in both directions and the confidence intervals did not have the correct coverage. With 10 or more events per variable, no major problems occurred. Vittinghoff and McCulloch (2007) found situations in which fewer than 10 events per variable gave acceptable results and concluded that the rule can be relaxed. Count events, not patients. Logistic regression is covered in more depth in the Practical Statistics chapter on logistic regression.

Time-to-event outcomes

What if the outcome is time until an event?

Cox regression models the hazard, the rate at which the event occurs at each point in time, from one or more predictors. Patients whose event has not happened by the end of follow-up (censored patients) still contribute.

hazard ratio for a 1-unit rise in x = e^b

D.R. Cox described the model in 1972. It assumes that the hazard ratio between groups stays the same over time (proportional hazards). A hazard ratio of 0.70 for a drug means that, at any time during follow-up, treated patients have the event at 70% of the rate of untreated patients. Survival curves, censoring and hazard ratios are covered in the risk and survival chapter.

Multivariate or multivariable

What does multivariate analysis mean?

Strictly, multivariate analysis models two or more outcome variables at the same time. A model with one outcome and several predictors is multivariable. The two words are often used as if they meant the same thing.

Hidalgo and Goodman reviewed one year of the American Journal of Public Health (December 2010 to November 2011). Of 30 articles that described a “multivariate” method, 25 (83%) had used multivariable analyses. They recommend describing a model as simple or multivariable for the number of predictors, and as linear, logistic, multivariate or proportional hazards for the type of outcome.

In textbooks and exam questions, “multivariate analysis” is also used as a broad heading for methods that handle many variables at once: multiple regression, MANOVA, principal component analysis, factor analysis, cluster analysis and discriminant analysis.

Ph.D. question

What is partial least squares regression?

Partial least squares (PLS) regression predicts an outcome from many predictors that are strongly correlated with each other, including data sets with more predictors than subjects. Ordinary least squares regression gives unreliable coefficients in the first situation and cannot be used in the second.

• When to use it: many correlated measurements, such as spectra, imaging data or large laboratory panels, often with few subjects.
• How it is checked: by cross-validation, which tests the predictions on data not used to fit the model.
• Origin: first proposed by Herman Wold in the 1960s, and widely used in chemistry (chemometrics) from the 1980s.
• Limitation: the components are combinations of many predictors, so the model predicts well but is harder to interpret one variable at a time.

In clinical studies

What do correlation and regression coefficients do in a clinical study?

Each coefficient answers a different question. The one you report decides what you can claim.

• Describing an association and generating hypotheses: r or ρ with a confidence interval.
• Controlling confounding in observational studies: adjusted coefficients, reported with the unadjusted ones (STROBE item 16a).
• Adjusting for baseline covariates in trials, planned in the protocol (ICH E9).
• Prediction: risk scores. In 1967 Truett, Cornfield and Kannel published the first multivariable risk function for coronary heart disease from the Framingham study.
• Calibration in the laboratory: a regression line of instrument reading on known concentrations converts a patient's reading into a concentration.

Decide

Which method fits your question?

Answer two or three questions. Each answer links to the section that explains it.

By specialty

Correlation and regression problems by specialty

Common myths

Common mistaken beliefs

• ✗Myth: r = 0 means the two variables are unrelated.

  ✓r = 0 means there is no straight-line relation. A U-shaped relation can give r = 0 exactly.

• ✗Myth: A strong correlation proves cause and effect.

  ✓Chance, reverse causation, bias and confounding can all produce a correlation. A causal claim needs a design that deals with them.

• ✗Myth: A high correlation between two methods of measurement means they agree.

  ✓Bland and Altman (1986) showed that correlation coefficients are misleading for this purpose. Two methods can correlate closely while one always reads higher. Plot the differences against the means. Methods for agreement are covered in the Practical Statistics chapter on ICC and Bland–Altman analysis.

• ✗Myth: A small p value means a strong correlation.

  ✓The p value depends on sample size. In 1,000 patients, r = 0.10 has p = 0.0015 and accounts for 1% of the variation.

• ✗Myth: The regression of y on x can be turned around to predict x from y.

  ✓In the worked example, SBP on age gives 0.58 mmHg per year. Age on SBP gives 1.17 years per mmHg, not 1 ÷ 0.58 = 1.72. The two lines coincide only when r is +1 or −1.

• ✗Myth: "Multivariate" means a model with many predictors.

  ✓That is a multivariable model. Multivariate means two or more outcome variables. In one year of a public health journal, 25 of 30 articles that said "multivariate" had used multivariable models (Hidalgo and Goodman, 2013).

Why this matters to you

Five consequences of misusing correlation and regression

1. Protocol sent back

   The analysis plan says "correlation and multivariate analysis" without naming the coefficient, the outcome or the predictors.

2. A causal claim the thesis cannot support

   A cross-sectional correlation written up as "vitamin D deficiency causes poor glycaemic control".

3. Viva questions on r²

   Unable to explain that r = −0.32 accounts for about 10% of the variation in the outcome.

4. Misreading papers

   Quoting an unadjusted odds ratio as the effect of an exposure, or accepting a logistic model with 12 predictors and 30 events.

5. Treatment that does not help patients

   Supplements prescribed on the basis of a correlation may not change the outcome at all.

History

History of correlation and regression

1. 1886

   Francis Galton

   Published “Regression towards mediocrity in hereditary stature”. Children of unusually tall or short parents were, on average, closer to the population average. He reported that the children's deviations were to those of their mid-parents “as 2 to 3”.

2. 1888

   Francis Galton

   In “Co-relations and their measurement, chiefly from anthropometric data”, defined co-relation between two measurements and showed that its closeness could be expressed by a single number.

3. 1896

   Karl Pearson

   Published a mathematical treatment of correlation and regression in the Philosophical Transactions of the Royal Society, calculating both from the product-moment. The letter r, which Galton had used for the slope of the regression line, came to stand for the correlation coefficient.

4. 1904

   Charles Spearman

   “The proof and measurement of association between two things” included comparison by rank, the basis of rank correlation.

5. 1958

   D.R. Cox

   “The regression analysis of binary sequences” dealt with outcomes recorded as 0 or 1 whose probability depends on one or more independent variables.

6. 1967

   Truett, Cornfield and Kannel

   Published the first multivariable risk function for coronary heart disease from the Framingham study.

7. 1972

   D.R. Cox

   “Regression models and life-tables” modelled the hazard as a function of explanatory variables and regression coefficients, multiplied by an unspecified function of time.

8. 1986

   J. Martin Bland and Douglas Altman

   Showed that correlation coefficients are misleading when two methods of clinical measurement are compared, and described a graphical alternative.

9. 1996

   Peduzzi and colleagues

   In a simulation study, logistic regression coefficients were biased when there were fewer than 10 events per predictor variable.

10. 2007

    Vittinghoff and McCulloch

    Found situations in which fewer than 10 events per variable gave acceptable results, and concluded that the rule can be relaxed.

Guidelines

Correlation and regression in reporting guidelines

Reading a paper

Five checks on any correlation or regression analysis

• Is there a scatter plot, and does the pattern suit the coefficient used (Pearson for a straight line, Spearman for ranks, skew or outliers)?
• Are r or ρ, its 95% confidence interval and n all reported, not only the p value?
• Does an observational study claim cause from a correlation?
• For adjusted estimates, are the confounders named with the reason for including each, and are the unadjusted estimates shown too?
• For logistic or Cox models, how many events were there per predictor variable, and were the model assumptions checked?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

Every TNMGRMU question on correlation and regression

Each one is answered in this chapter. Tap a sitting to open the university's own question paper.

1. ×4Regression analysisMD Biochemistry · Oct 2023MD Nuclear Medicine · Jul 2021DM Neonatology · Feb 2020MD Biochemistry · Apr 2016
2. ×4Correlation coefficientMD Nuclear Medicine · May 2022MD Nuclear Medicine · May 2018MD Community Medicine · May 2017Ph.D. RM · Jul 2014
3. ×2Multivariate analysisPh.D. RM · Jul 2022Ph.D. RM · Jul 2014
4. ×1Write two methods of correlation commonly used in research methodologyMD Biochemistry · May 2024
5. ×1Correlation coefficient and Pearson product-moment correlation coefficientDM Neonatology · Feb 2022
6. ×1What is correlation coefficient? What is its significance?MD Biochemistry · May 2017
7. ×1Explain regression and correlation in clinical studies with appropriate examplesPh.D. RM · Jul 2017
8. ×1Correlation and RegressionPh.D. RM · Jul 2015
9. ×1(a) Explain what is meant by ‘Multivariate Analysis’. (b) Explain in detail Partial Least Square Regression. (c) Explain what is Meta Analysis, the steps of Meta Analysis, the methods and assumptions of Meta AnalysisPh.D. RM · Jul 2013
10. ×1Correlation coefficient and coefficient of variationMD Biochemistry · Sep 2009
11. ×1What is the role of correlation and regression coefficient in statistical analysis?MD Community Medicine · Sep 2009
12. ×1Logistic regressionDM Neurology · Aug 2009

Further reading

Related Stateazy articles