Preparing Data in Stata
The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.
At 10:17, the video shows a do-file that does not recode the negative codes in b02 to missing. Respondents who refused to say or did not know how they voted (codes −2 and −1) are therefore counted as “Another Party”. The do-file now includes replace b02 = . if b02 < 0 in Operation 1.
You’ll learn to
Depending on what dataset you are working on and what variables you choose for your analysis, you might need to prepare your data before you start to analyse it. In this video you will learn some of the tools that allow you to do that. You will learn to:
- Recode invalid values to missing using
replace. - Drop observations based on a condition.
- Generate a binary variable from a logical statement.
- Save a cleaned dataset as a separate file.
Materials
The companion do-file for this video is prepare-bes.do, which works with the British Election Study. It is available in the Shared Project Folder, in the do folder. Save it into the do folder on your own computer.
Why data preparation matters
Raw data can contain invalid or missing value codes, and analysing uncleaned data can, depending on the situation, produce incorrect results. A short preparation step before the analysis prevents these errors.
An important preamble
Data preparation is not always necessary. Depending on the data you choose, you might do just as well analysing the raw data: it depends on how the dataset you are working with, and its variables, are coded.
It is important for you to know that the examples here exist. You should then look at the variables in the codebook and how they are coded, consider whether any of these commands will be necessary, and be able to adapt them to the specific task: to the data you are using and the variables you are going to use in your analysis.
So you should learn to exercise your judgement to work out when data preparation is needed, what commands to use, and how to use them. You can always ask for help with this, but it is a skill that is important for you to learn.
Missing values in Stata
Suppose there is an observation in your data for which a piece of information is not available: for example, someone in the British Election Study who refused to tell the survey enumerator their age. In Stata, a missing value is represented by a dot, ..
Sometimes, though, survey datasets like the British Election Study code those missing responses as negative numbers. Take the age variable in the British Election Study codebook. When respondents were asked “Please can you tell me your age at your last birthday?”, some people refused, and in the dataset that shows up as −2.
If you calculate the mean age of the people in the British Election Study without recoding those −2s to dots, that is, to missing values, you get a mean that is calculated under the assumption that some people’s age is −2. The −2s of those who refused are computed into the mean. You underestimate the mean age and end up with an invalid mean, because of those missing values being coded as negative numbers. These codes must be recoded to . before analysis.
Operation 1: Recode invalid values to missing
This is how you replace them:
replace age = . if age < 0The command says: replace age with a dot if age is less than zero. It tells Stata to replace those negative numbers with dots, which represent missing values. Stata then excludes those observations from calculations, so when you ask for the mean of age you get the actual mean age. Repeat this for each variable that uses negative codes for missing data.
To see this in the do-file, load the dataset and summarise the age variable. Note that in the dataset the variable is spelled Age, with a capital A.
summarize AgeThe mean is 51.7. But when you look at the minimum, it is −2. These −2s are being computed into the mean. So we run the replacement:
replace Age = . if Age < 0Stata reports that 60 real changes were made, 60 to missing. If you summarise Age again, the minimum is now 18, and the mean is not 51.7: it is actually 52.8. Notice also that the number of observations decreased by 60, from 3,089 to 3,029, the 60 that went to missing, as a result of replacing those negative values with dots.
Operation 2: Drop observations based on a condition
A second type of operation is to drop observations based on a condition. This, too, is not going to be applicable in every case. But suppose we are testing a hypothesis about whether Mail and Telegraph readers have different ideologies. If we are only interested in people who read either the Daily Mail or the Daily Telegraph, then we want to drop certain observations from the data. drop if does that.
drop if k03 != 7 & k03 != 13Here k03 is the newspaper variable, 7 is the Daily Mail and 13 is the Daily Telegraph. The exclamation mark followed by an equals sign means “is not”. So the command says: drop the observation if the newspaper is not the Daily Mail and is not the Daily Telegraph. It restricts the sample to the two groups being compared.
The equivalent using keep if
keep if works as an equivalent. One command tells Stata to drop observations based on a condition; the other tells it to keep observations based on a condition.
keep if k03 == 7 | k03 == 13This achieves the same result. The drop if version says: drop if the newspaper is not the Daily Mail and is not the Daily Telegraph. The keep if version says: keep if the newspaper is the Daily Mail or the newspaper is the Daily Telegraph. It is important to be able to work with these logical statements, so that you know what conditions to state.
In the do-file, the keep if line is commented out. When a line starts with a star, it is a comment and Stata will not run it. Remove the star, select the line, and Stata will run it.
Running the drop if version, Stata reports 2,821 observations deleted. If you reload the dataset and run the keep if version instead, you delete the same number of observations, 2,821, because these are equivalent commands.
Operation 3: Generate a binary variable
A third operation is to generate a binary variable: a variable that takes on only two values. For example, when we test whether men are more likely to vote Reform than women, we generate a new variable called voted_reform. There is a variable that tells you the party each person voted for, but for the purposes of that comparison we are only interested in whether or not they voted Reform.
gen voted_reform = (b02 == 8) if b02 != .The expression (b02 == 8) returns 1 if the condition is true and 0 if it is false. The 8 here means Reform: when you go to the codebook and look at how each party is coded, observations coded with the number 8 are people who voted Reform.
The final part, if b02 != ., means “if b02, the party variable, is not missing”. This makes sure that missing values are not misclassified as not having voted Reform. These are people for whom we do not know who they voted for, so we cannot say that they did not vote Reform. For this to work, the negative codes in b02 must first be recoded to missing, as in Operation 1: replace b02 = . if b02 < 0.
So the command tells Stata: generate a voted_reform variable that is 1 if the condition is true and 0 if it is false, but only do this if b02, the party the person voted for, is not missing. You use this when you need to reclassify multiple categories into two.
Labelling the new variable
The next commands label the new variable, which makes output easier to read without changing the underlying values.
label define reform_lbl 1 "Reform" 0 "Another Party"
label values voted_reform reform_lbllabel define creates the set of value labels: an object that stores the information that 1 is “Reform” and 0 is “Another Party”. label values attaches that set of value labels to the new variable.
Operation 4 (Optional): Save the cleaned dataset
Something that is optional is to save the cleaned dataset as a new file in your data-clean folder.
save "data-clean/bes_clean.dta", replaceYou do this so that you do not overwrite the raw data. You keep the original, raw data where it is, and if you want to save the modified version of the data, you save it as a new file in a different folder. Here we call it bes_clean.dta. The replace option overwrites any previous version of the clean file.
If you are analysing the British Election Study, you might not necessarily want a dataset with exactly this recoding. Remember that these commands are not one-size-fits-all. But this is something that you can do, and you can then use the new dataset that you saved to run your analysis.
Recap
replace x = . if x < 0recodes invalid values to missing.drop if conditionremoves observations outside the subgroup of interest. Remember that in most cases you will not want to drop everyone who does not read the Daily Mail or the Telegraph: that command was specific to the intention of comparing the ideologies of those subgroups.generatewith a logical statement creates a binary variable, a variable with only two values, which is something you might need for your analysis.savestores the cleaned data in a separate folder, so that you never overwrite the raw data.
Why this matters
Uncleaned data produces unreliable results. Even correct commands will fail if the inputs are wrong, as we saw with the mean of the age variable being miscalculated because of the −2s in the dataset. Good data preparation habits make your work transparent, reproducible, and easier to check.
But remember: it is important to know that the commands you learned here exist, yet you might not need them. You might not need any of them in your analysis. Part of the skill you need to develop is exercising the judgement to look at the codebook, see how the variables are coded, see what analysis you need, and decide whether or not you need to recode anything. That skill is important for data analysis.