Multivariate Regression

The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.

You’ll learn to

So far, you have been learning about bivariate regressions: regressions with two variables, one dependent variable and one independent variable. Now you are going to start learning about multivariate regressions, which have one dependent variable and multiple independent variables. In this video you will learn to:

  • Write a regression equation with multiple explanatory variables, that is, multiple independent variables.
  • Interpret a slope “holding other variables constant”, which is what we do when we analyse the results of multivariate regressions.
  • Compare bivariate and multivariate regression results.

Recall: Education and the Leave vote

Let’s get back to the Brexit example. We have seen that the share of Leave votes differs between regions of the UK depending on the percentage of people in each region with higher education. When we regress Leave vote share, as the dependent variable, on the percentage with higher education, the independent variable, we get a slope of \(b = -1.13\). One more percentage point in the share with higher education corresponds to a drop of 1.13 percentage points in the share of Leave votes.

The R-squared is \(R^2 = 0.30\), so education explains 30% of the variation in the share of Leave votes. And this is a statistically significant slope: we reject the null hypothesis that there is no relationship between the two variables when we run this bivariate regression.

Another predictor: Immigration and the Leave vote

Another predictor, another independent variable that can form a separate bivariate regression, is related to immigration: the share of people in a region who are immigrants, by region of the UK.

When we take the Leave vote share as the dependent variable, on the y-axis, and the immigrant share as the independent variable, on the x-axis, we get a slope of \(b = -0.41\). It is also statistically significant (\(p < 0.001\)). So we do see this correlation, where regions with more foreign-born residents voted Leave less.

So you have the possibility of running these two separate, distinct bivariate regressions. One is a regression model, and the other is a different regression model with a different independent variable.

Can we include both variables at once?

Now we can include both variables at once. When we look at those two bivariate regressions, it looks like both education and immigration predict the Leave vote separately. But we can put them in the same model. We can run a regression with Leave share as the dependent variable and both explanatory variables: the share with higher education and the share of immigrants in regions of the UK. This is what a multivariate regression does.

The multivariate regression equation

A simple regression, or bivariate regression, has one explanatory variable. You have the predicted value of Y as a function of the intercept, identified here as \(b_0\), plus \(b_1\) times X:

\[\widehat{Y} = b_0 + b_1 X\]

A multivariate regression has two or more explanatory variables. Now you have the predicted value of Y, Y hat, being the intercept plus \(b_1\) times \(X_1\) plus \(b_2\) times \(X_2\):

\[\widehat{Y} = b_0 + b_1 X_1 + b_2 X_2\]

You are adding a variable to the right-hand side of the regression equation.

Our multivariate regression

We can then run the multivariate regression. This is the result when we have Leave vote share as the dependent variable, and education and immigration as independent variables.

Coefficient P-value
Education -1.14 < 0.001
Immigration 0.08 0.449
Intercept 83.49 < 0.001

\(R^2 = 0.30\), \(N = 139\).

Interestingly, when you do this, the coefficient for education is almost the same as before, and it is still statistically significant. The coefficient for immigration, however, is starkly different. In the multivariate regression it is close to zero, and it is not statistically significant. And the R-squared is very similar to the R-squared from the regression of Leave share on education.

Subsequent videos discuss what this means substantively. Here, let’s keep to the basics: what the multivariate regression does, and the new phrase that you are going to learn to say when you are interpreting a multivariate regression, which is to hold something constant.

Interpreting “holding constant”

Now that we have a multivariate regression, we can say that the education slope, −1.14, means that when you increase the share of people with higher education by one percentage point, you expect the share of Leave votes to decrease by 1.14 percentage points. But now you have to say that you are holding constant the level of immigration. You are comparing regions with similar immigration. Once you add a new variable, once you are talking about a multivariate regression, this is the correct way to interpret the slope. “Holding constant” isolates one variable’s contribution.

An equivalent phrase is “all else equal”: everything else remaining the same.

The equivalent for immigration would be: holding constant the level of education, a one percentage point increase in the share of immigrants corresponds to an increase of 0.08 percentage points in the share of Leave votes. But notice that here we are not looking at a statistically significant relationship. When we are not holding education constant, when we are not comparing regions with similar levels of education, the bivariate regression suggests a relationship between immigration and the Leave vote. Once you account for education, once you start comparing between regions with similar levels of education, you no longer see a relationship between immigration and the Leave vote share.

Again, the substantive interpretation is discussed in the next video.

What changed?

  • The education slopes are almost identical: −1.13 in the bivariate regression, −1.14 in the multivariate one.
  • Immigration is significant alone (−0.41), but not in the multivariate model (0.08). It is no longer significant when we are looking at levels of immigration across regions with similar education levels.
  • The \(R^2\) barely changed.

Recap

  • A multivariate regression includes more than one explanatory variable.
  • Each coefficient is interpreted “holding other variables constant”: holding constant the other variables included on the right-hand side of the regression equation.
  • Coefficients can change when additional variables are included. The immigration coefficient changed radically going from the bivariate regression of Leave share on immigration to the multivariate regression.

Why this matters

Most outcomes in the social sciences have multiple causes, and a single explanatory variable is rarely going to tell us the entire story. Multivariate regression allows us to examine how those different variables contribute to the outcome.