Inference in Regression: CIs and P-values

The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.

You’ll learn to

Just as differences in means and differences in proportions have confidence intervals and p-values, the same logic applies to the regression slope. The slope also has confidence intervals and p-values built around it, so that we can use a regression to make inferences: to go from data from a sample to conclusions about a population. In this video you will learn to:

  • Translate a verbal hypothesis into a regression hypothesis, so that regressions can be used to test hypotheses.
  • Define the p-value for a slope estimate.
  • Use a confidence interval for the slope to test the null hypothesis.

From description to inference

Let us go back to the equation: the predicted Leave vote share in the Brexit referendum as a function of the percentage of people with higher education in any given region of the UK.

\[\widehat{\text{Leave}} = 84 - 1.13 \times \text{HigherEd}\]

We estimated a slope of −1.13. However, we are working with an estimated slope, and there is also a slope out there in the world that actually describes this relationship. The underlying premise is that the real slope could be zero, and the −1.13 that we see when we estimate this regression could be the result of chance variation in the data rather than a real relationship between the share of people in each region with higher education and the share of Leave votes in the Brexit referendum.

So how do we test whether the slope is statistically significant? This is another term that comes up a lot: whether an estimate is statistically significant.

Stating the hypothesis in regression terms

\[\widehat{\text{Leave}} = \hat{\beta}_0 + \hat{\beta}_1 \times \text{HigherEd}\]

  • Alternative hypothesis (H1): regions with more graduates voted Leave less.
  • In regression terms: \(\hat{\beta}_1 < 0\).

When we say that regions with more graduates voted Leave less, we are postulating a negative relationship, and therefore, in regression terms, a negative slope.

The null hypothesis

  • Null hypothesis (H0): education share is unrelated to Leave vote share.
  • In regression terms: \(\beta_1 = 0\).

The null hypothesis is that the education share has no relationship to the Leave vote share. In regression terms, this means that the slope is zero. So the null hypothesis that we are testing against when we build confidence intervals and look at p-values is that the slope is zero.

What H0 = 0 means

If \(\beta_1 = 0\), if the slope is zero, the regression line is flat. Higher education would have no predictive value for the Leave vote share: no matter how much the percentage of people with higher education goes up and down, there would be no associated difference in the Leave vote share.

Under this null hypothesis that the real slope is zero, the estimated slope of −1.13 would be due to chance variation in the data. The null hypothesis states that the −1.13 we get when we calculate the regression is not capturing a meaningful relationship between the share of people with higher education and the Leave vote share.

The significance test for the slope

We estimated \(\hat{\beta}_1 = -1.13\). What we want to know is whether this is far enough from zero to rule out chance as the reason we are getting that −1.13.

The p-value for the slope

The slope has a p-value, which gives us a sense of how confident we can be that the −1.13 is not zero.

Remember the general definition: the p-value is the probability of observing data at least as extreme as the data we are observing, assuming that the null hypothesis is true. In the context of a regression, the p-value is the probability of observing a slope at least as extreme as −1.13 assuming that the true slope is zero, that is, assuming that the null hypothesis is true. A small p-value means that the result is unlikely to be a chance finding.

Note. Saying that a small p-value means the result is “unlikely to be a chance finding”, or that it tells us “how confident we can be” that the slope is not zero, is a simplification. More precisely: a small p-value means that a slope this far from zero would be unlikely if the true slope were zero. It is not the probability that the result is due to chance.

The decision rule

Remember the significance level. Social scientists usually work with a significance level of 5%, an alpha of 0.05: \(\alpha = 0.05\).

  • If p < 0.05, we reject the null hypothesis. This means that the slope is statistically significant.
  • If p > 0.05, we fail to reject the null hypothesis. We conclude that the data are not giving us enough evidence to reject it.

Confidence intervals for the slope

Remember that p-values and confidence intervals are different ways of testing hypotheses, but they never contradict each other. So we can also construct a 95% confidence interval for the slope, for \(\hat{\beta}_1\).

  • If the confidence interval does not include zero, we conclude that zero is not a plausible value for the slope, and we reject the null hypothesis.
  • If, on the other hand, the confidence interval includes zero, we fail to reject the null hypothesis, the null hypothesis being that the slope is zero.

Brexit results: applying the logic

Applying the logic to the Brexit example:

  • The estimated slope is \(\hat{\beta}_1 = -1.13\), which is negative, as the alternative hypothesis, H1, predicts.
  • When we calculate the regression, we get a 95% confidence interval for the slope of [−1.42, −0.84]: the lower bound is −1.42 and the upper bound is −0.84.
  • The p-value is below 0.05.

Brexit results: the decision

The 95% confidence interval is entirely below zero, so it does not include zero, and the p-value is smaller than 0.05. A p-value smaller than 0.05 means that it is highly unlikely that we would observe these data if the null hypothesis were true.

So we reject the null hypothesis and conclude that regions with higher education shares had a systematically lower Leave vote share. The result is highly unlikely to reflect chance variation alone.

What makes a result more significant?

When we estimate those 95% confidence intervals and p-values, what makes a result statistically significant?

  • Larger sample: the larger the sample, the smaller the p-value.
  • Steeper slope: the further the slope is from zero, the smaller the p-value.
  • Less scatter around the regression line also gives you a smaller p-value.
  • Greater spread in x gives you a smaller p-value.

Holding everything else constant, if you increase the sample, get a steeper slope, have less scatter around the regression line, or have a greater spread in x, you get a smaller p-value.

Recap

  • If the alternative hypothesis, H1, says that the slope is different from zero, \(\beta_1 \neq 0\), then the null hypothesis, H0, says that \(\beta_1 = 0\).
  • The p-value is the probability of a slope at least as extreme as ours, if H0 were true.
  • A 95% confidence interval excluding zero is equivalent to a p-value smaller than 0.05. The two decision rules never contradict each other.

Why this matters

Every regression table that you read reports the slopes and the p-values. It reports whether the result is statistically significant, which is something important to take into consideration when you are interpreting statistical results. This video gives you the logic that you need to interpret these outputs correctly.