Interpreting P-values
The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.
You’ll learn to
In the previous video, you learned about testing hypotheses using confidence intervals. Now you will learn about a related and equally important concept for hypothesis testing: the p-value. In this video you will learn to:
- Interpret a p-value and explain what it measures.
- Use a p-value to evaluate a null hypothesis.
A question to keep in mind
In the previous videos we were testing hypotheses using confidence intervals. P-values offer a different way to answer the same question. The underlying question is always: how surprising are our results if the null hypothesis were true?
Example: Is a coin fair or loaded?
Let us work through the example of tossing a coin when we do not know whether the coin is fair or loaded. It could be a coin like any other: if you toss it, you have a 50% probability of heads and a 50% probability of tails. But a coin could also be loaded, weighted in such a way that it has a much higher probability of landing heads up, for example.
Suppose we have a coin, we do not know whether it is fair or loaded, and we want to check. To check, we toss it multiple times and see what we get: with some results we would say it is probably loaded, with others that it is probably fair. So we flip the coin over and over again and register the results.
- Null hypothesis: the coin is fair. Heads and tails are equally likely.
- Alternative hypothesis: the coin is loaded. Heads are more likely than tails.
The alternative hypothesis is the hypothesis that we are trying to test.
The logic of the test
The logic of the test is that we flip the coin, we observe the result, and then we ask ourselves: assuming the coin is fair, how likely is this outcome? We collect data on every single coin toss.
If the observed result would be very unlikely under the null, that is evidence against it. But if what we are seeing is not unlikely, we have no strong reason to doubt the null hypothesis.
Building the evidence
Imagine that we flip the coin five times and every single time we get heads: the first time heads, the second time, the third time, over and over again.
| Flips | All heads? | Probability if coin is fair |
|---|---|---|
| 1 | Yes | (½)¹ = 50% |
| 2 | Yes | (½)² = 25% |
| 3 | Yes | (½)³ = 12.5% |
| 4 | Yes | (½)⁴ = 6.3% |
| 5 | Yes | (½)⁵ = 3.1% |
When we flip it the first time and it is heads, we have no reason to conclude that it must be loaded, because there is a 50% probability of a fair coin landing heads up. If we toss a fair coin twice and both times it is heads, there is still a 25% probability of this outcome. It is not entirely unlikely; it could happen with a fair coin. With a third heads the probability drops to 12.5%, with a fourth it drops to 6.3%, and with a fifth it drops to 3.1%: that is the probability of flipping a coin five times and getting heads every single time.
Each consecutive heads makes the fair-coin explanation less plausible. By the fifth toss, we are thinking that this is probably a loaded coin.
What is a p-value?
The probabilities in that table are p-values. This is the definition: the p-value is the probability of observing a result at least as extreme as the one we got, assuming the null hypothesis is true.
A small p-value means that, assuming the null is true, what we are seeing is highly unlikely. A large p-value means that the data are consistent with the null: we have a reasonable probability of observing results like these, assuming that the null hypothesis is true.
How small is small enough?
How small a probability is going to get us to decide? In quantitative social science we have the convention of using 5%, or 0.05, as a threshold. This is called the significance level, or α (alpha). It is an important concept for testing hypotheses. Most of the time, when we look at hypothesis tests in quantitative work in social science, people are working with a significance level of 0.05, a 5% threshold.
Applying the threshold
| Flips | All heads? | P-value | Decision (α = 0.05) |
|---|---|---|---|
| 4 | Yes | 6.3% | Fail to reject the null |
| 5 | Yes | 3.1% | Reject the null |
A quantitative social scientist looking at the coin flips would still fail to reject the null when it gets to the fourth coin toss and it is heads. But the fifth time around it is still heads, and the probability is lower: the probability of observing that result with a fair coin is below 5%. It falls below the threshold, so we reject the null.
At four heads, the result is unusual but not unusual enough. At five heads, it is too unusual, and we reject the fair-coin hypothesis.
This is a convention. The 5% is not a magical number. It is something that researchers have conventionally agreed to use as a threshold for testing hypotheses. It might seem like a magical thing, but the best that we have as a collective of scientists is to use a convention, and 5% is the most common significance level, the most common threshold.
P-values and confidence intervals
In the previous video we were testing hypotheses with confidence intervals. P-values ask the same question differently. However, the two approaches, testing hypotheses with confidence intervals and testing hypotheses with p-values, will never contradict each other.
Why they always agree
Whenever you run a hypothesis test, the results are usually reported with both the 95% confidence interval and the p-value. Thinking back to the gender and Reform example, you would see a p-value there and you would see a confidence interval.
- A 95% confidence interval excludes zero exactly when the p-value is smaller than 0.05 (p < 0.05).
- A 95% confidence interval includes zero exactly when the p-value is equal to or larger than 0.05 (p ≥ 0.05).
These are two windows onto the same underlying calculation. They are not two independent tests. Later you will learn how we get those two numbers, but for now it suffices to understand that they will never contradict each other. They are calculated using a key statistic, the standard error, which you will learn about.
What a p-value is not
A p-value is not the probability that the null hypothesis is true. The p-value is the probability of observing that pattern, of observing those results, under the assumption that the null hypothesis is true. These are different things. A small p-value means that the data would be unusual if the null were true.
Recap
- The p-value is the probability of observing results at least as extreme as ours, assuming the null hypothesis is true.
- P-values and confidence intervals always lead to the same conclusion: they are two ways of expressing the same hypothesis test.
Why this matters
You will see p-values in almost every quantitative study in political science and in social science. Much like confidence intervals, p-values are ubiquitous in quantitative research, and reading them correctly is a basic requirement for evaluating evidence critically.