Why Sampling Matters

The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.

You’ll learn to

So far in this video series we have been looking at descriptive statistics, which describe the data from samples. Soon we will start learning about drawing inferences from those samples to an entire population. In this video you will learn to:

  • Define a population and a sample.
  • Explain how we use samples to draw conclusions about populations.
  • Explain why how data is collected affects what conclusions we can make.

We want to know about populations

Once we have a research question, we want to collect data. Ideally, we want data on every single unit we are interested in. In the case of the British Election Study, for example, we would ideally want a study that could interview every single person in the United Kingdom.

This full set of units is called the population. Every single voter in the United Kingdom: that is the population we draw inferences about when we work with the British Election Study.

Populations depend on the question

The population is defined by who or what we are studying. The trick here is that the word population, in the lingo of statistics, does not necessarily mean people. It might mean people, but not necessarily. The population could also be all countries in the world. It could be all cities in the United Kingdom. It could be all wars.

Whatever the thing you are studying, there is going to be a population and a sample. Populations are whatever units our question is about.

The problem: populations are often inaccessible

The problem is that getting data on every single unit of the population is often impossible. It can be too expensive, too slow, or just unfeasible. We need a way to learn about the population without collecting data on every single unit.

The solution: samples

This is where samples come in. A sample is a subset of units drawn from the population. We collect data on the sample and use it to draw conclusions about the entire population.

One famous statistician, Gary King, defines this process of inference from a sample to a population as using the facts that we have to learn about facts that we do not have. That is statistical inference. The facts that we have are the facts in the sample, and we use them to learn about facts that we do not have, which is information about every single unit in the population.

This is, for example, what opinion polls do before an election. They collect data from a sample; they do not interview every single voter. But from that sample we can draw conclusions about the chances of each candidate or each party winning the election, because we are using the facts that we have from the sample to learn facts that we do not have about the population.

From sample to population: inference

That is the definition of inference: drawing conclusions about a population using a sample.

The thing about sampling is that we can only make inferences from certain types of samples. Not every sample gives us the ability to use it to draw conclusions about the population. So how the data is collected is very, very important.

Not all samples are equal

Imagine a news outlet that polls only its own viewers. Take the example of ideology and the newspapers that people read. Imagine a poll that drew only from the readers of one newspaper, which has a certain political or ideological leaning.

If the viewers or readers of a news outlet skew towards one political side, the sample will over-represent those types of viewers, and it will contain more voters who would vote in the direction of that political side. The sample would not look like the population, and any inference we tried to draw from it would be completely misleading.

Non-probability samples

This is an example of a non-probability sample, which means that not every unit in the population has a chance of being selected. These samples tend to be unrepresentative. They tend to be biased, and they are not trustworthy for drawing inferences about the population. We cannot use these types of samples to draw inferences.

Probability samples

By contrast, in a probability sample every unit has a known chance of being selected. For an opinion poll, you need to collect the sample in a way that gives every single person a known probability of being selected.

There are several ways to achieve that. The most important and most illustrative is the simple random sample. Imagine that we had the names of all registered voters in a dataset, and a computer drew, at random, say 2,000 to 3,000 people. That would be a simple random sample: every unit has an equal probability of being chosen.

The opinion polls we see do not do exactly a simple random sample, but they try to approximate it to the best of their ability. The process just described, drawing all of those names and then knowing how to contact each person, raises ethical issues and practical issues as well. So the simple random sample is not exactly what opinion polls do, but pollsters do their best to approximate it.

Why random samples are powerful

Random samples produce samples that are representative of the population, samples that are unbiased. In all likelihood, the political leanings and preferences of the sample will not be identical to those of the population, but they will be close enough that we can draw inferences and learn about the population from the sample.

The characteristics of the sample will closely mirror those of the population. They will not be identical, but they allow us to generalise from the sample to the population with some degree of confidence.

Recap

  • A population is the full set of units we want to draw conclusions about.
  • A sample is a subset of the population.
  • Inference means using the facts that we have, from the sample, to learn about facts that we do not have, about the population.
  • Random samples are representative and support valid inference; non-probability samples do not. More precisely, it is probability samples, including but not limited to random samples, that can support valid inference about the population.

Why this matters

Almost all political science data, and almost all social science data, comes from samples, not full populations. Understanding how a sample was collected determines what conclusions we can draw.