Variance and Standard Deviation

The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.

You’ll learn to

In this video you continue to learn about gathering additional information about a distribution, so that we can put the numbers in context. You will learn to:

  • Interpret the variance and standard deviation as summaries of dispersion.

How good is 80%?

Remember the example of the course with the 80% approval rate. A university advertises that 80% of students are satisfied with its course, according to the National Student Survey (NSS). But the national mean is about 82%. With this new information, when you look at the 80% you realise that this approval rate is not as impressive as it might look in the beginning.

Variance and standard deviation

When we add the variance and the standard deviation to the mix, we gain additional context: new information that allows us to further investigate how typical or how unusual an observation is. They allow us to observe the extent to which an observation stands out from the rest, for good or for bad.

Variance

Let’s go through, step by step, how you calculate the variance. The variance measures how far from the mean the observations tend to be. Remember that to get the mean you sum the values of all observations and divide by the number of observations.

The mean here is 82%, so the course with an 80% approval rate is not too far from the mean. But how about the others? To calculate the variance we have to look at all of the courses in the data, all of the observations in the distribution.

Calculating the variance

To get the variance, we start by subtracting the mean from each observation and squaring the difference. So we take the value of each observation minus the mean, squared, and then we start to sum these up.

Let’s start with a concrete, real approval rate. Linguistics at the University of Glasgow has a 90% approval rate. For that particular course, we take the approval rate minus the mean and square the difference:

\[(90-82)^2=8^2=64\]

We then sum that 64 with the squared deviations of every other course in the NSS data. We do this for every course and take the sum of the squared deviations for all courses. When we divide this sum by the number of courses, we get the variance.

Any given course that is far from the mean increases the variance. Squaring is important here because if a course is below the mean, the difference is a negative number. We square it so that those negative numbers do not cancel out the positive numbers. If a course is far from the mean but far below it, that still counts in a way that increases the variance when you do the calculation.

The variance formula

This is a formula with a lot of Greek letters that looks scary, but it only describes the step-by-step procedure that we just went through.

\[\sigma^2 = \frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}\]

Once you understand what each Greek letter means, you will see that this is just a mathematical form of that procedure.

  • \(\mu\) is the mean. It is 82 in the case of the NSS approval rates.
  • \((x_i - \mu)^2\) is what we did earlier. Using the example of linguistics at the University of Glasgow, \((90-82)^2\) is equivalent to this part of the formula.
  • \(\sum_{i=1}^{N}\), the capital sigma with the \(N\) on top and the \(i=1\) at the bottom, means “do this for every observation (\(x_i\)) and take the sum.” The thing that we just did for the University of Glasgow, do that for every course in every university and then sum it all up.
  • We divide it by \(N\) to get the variance (\(\sigma^2\)), the lower-case sigma squared.

So the Greek letters are just describing the step by step from before.

The standard deviation

The standard deviation is the square root of the variance. It is more commonly used than the variance, and it is especially useful to judge how unusual an observation is. We will go through a couple of examples so that you can see how to use the standard deviation.

You do not need to calculate the variance by hand: any statistical software will do it for you. When you calculate the variance for satisfaction with courses in the National Student Survey, it is about 145. Then you take the square root of that and you get the standard deviation, about \(\sqrt{145}\approx12\). So the standard deviation for approval rates is approximately 12.

Interpreting the standard deviation

How do you interpret that? Let’s look at the linguistics course at the University of Glasgow, with its 90% approval rate. It is eight points above the mean, and those eight points correspond to two-thirds of a standard deviation:

\[8/12=2/3\textrm{ s.d.}\]

So linguistics at Glasgow is two-thirds of a standard deviation above the mean. This signals that it stands out to some extent: it is a decent deviation above the mean. For people who understand data, the linguistics course at Glasgow can advertise its approval rate with more confidence than by just throwing the 90% out there in isolation.

A different course, accounting at Strathclyde, can be even more boastful, because it is one standard deviation above the mean. Its approval rate is 94% and the mean is 82:

\[12/12=1\textrm{ s.d.}\]

That is one entire standard deviation above the mean, so it stands out from the mean more than linguistics at Glasgow does.

Now think about different distributions, for example annual income in London. There you have values on the scale of tens of thousands of pounds. To say that someone earns so many pounds above the mean tells you something. But when you calculate the standard deviation and say that someone earns one standard deviation above the mean, that tells you more about how much that person’s income stands out from the distribution of income in any given place.

Recap

  • Variance and standard deviation summarise how spread out a distribution is.
  • They are built by measuring how far observations are from the mean.

Why this matters

The variance and the standard deviation matter because they add context to our understanding of a distribution, by telling us whether the differences from the mean are small or substantial. Knowing the mean tells us what is typical. When you have the mean, the median and the standard deviation, you have a good toolkit for assessing whether an observation is more typical or more unusual.