Doing Reproducible Research
The text below is a transcript of the video above, edited with AI assistance for clarity and flow. Where the two differ, the video is the authoritative version.
You’ll learn to
Knowing how to set yourself up for conducting reproducible research is a core skill in quantitative analysis. In this video, you will learn to:
- Explain what reproducible research means and why it matters.
- Set up a project folder structure on your computer that supports reproducible work.
What is reproducible research?
What do we mean when we bring up this concept? Reproducible research means being able to obtain the same results using the same data and code. That applies to anyone who gets hold of the data and the code that you used to produce a study or a statistical analysis. Anyone else should be able to follow the steps of your analysis and arrive at the same results that you arrived at. That includes you in a few weeks’ time. So it must be reproducible by you and by others.
Why care about reproducibility?
Why is reproducibility an important thing in quantitative research? Whenever we are analysing data, we are always making incremental changes. Those small changes accumulate quickly, and we often forget what we are doing and why we are doing it.
When we have clear structures, a folder structure with the right files in the right places, it becomes easier to detect and fix mistakes. That is something you yourself will be able to do, and so will others when they are trying to reproduce your analysis. This kind of organisation, which allows you in the future and others to retrace your analysis and detect and fix mistakes, is important both for the transparency of research and for the credibility of quantitative research more broadly.
Data and code
The data and the code are completely different things, and both are integral parts of any analysis. They are the building blocks of quantitative analysis.
The data is the information that you gather to begin your analysis. For example, you might gather data from public opinion surveys, election results or national census statistics in order to conduct research, test your hypotheses and design your study.
The code is where you establish the instructions, the actions that transform and analyse the data. For example, the code tells the software you are using, which could be Stata or could be R: merge this data on economic growth rates with election results, and now create a table that shows how many elections were held in recession years. All of these instructions, these tasks, these actions, go in the code.
For reproducible research, it is critical to always keep the data separate from the code. It is not always the case that research separates them. Sometimes research is conducted in a way where the information and the instructions are merged together, and you do not know which instructions did what in a study or how someone arrived at those results. That is not good practice. We want to be able to reproduce research, and to do that we must separate the data from the code.
What this looks like in Stata
What is this going to look like when you are doing a statistical analysis in Stata?
The dataset that you download is treated as a read-only file. It is a file that you will not modify. If you downloaded the national census from a government website, you keep that data exactly as it looks at the source where you downloaded it. You do not change it at all. You read from it, and you ensure that someone else who goes to the same website and downloads the same data will have the same file to work with.
You could transform the data, for example by merging it with something else, and you could analyse it, but all of these changes happen in a Stata-specific file that is called the do-file. The do-file keeps a record of everything that you do with the data, including transforming it and analysing it.
A simple project folder structure
You need to separate the folders where each of these components of your research project is stored. There are many ways to set up folder structures for reproducible analysis. I suggest that you start with something simple: a few folder names and roles to begin with. You can always add more folders as you move on, as you see fit.
Say you have a folder where you are going to conduct a research project. I came up with the generic name my-research-project. This is the folder where everything happens. Inside that folder, I suggest that you have four folders:
my-research-project/data-rawdata-cleandooutput
These are the very basics. What is the role of each folder, and what does each one contain?
data-raw- Original data files, never edited.
data-clean- Cleaned or transformed datasets.
do- All Stata do-files.
output- Tables and figures.
The data-raw folder is where the original data files go: the files that you are never going to edit, the files that you leave as read-only. You only extract information from them, never changing what the data are at the source. If you download something from a government website or from a public opinion survey, you leave the data you download in your data-raw folder. You use the software to pull information from that file, but you do not change it at the source.
Any transformations that you make to your data, you save as a new file in a different folder, called data-clean. That is the folder for cleaned or transformed datasets.
The do folder is where we keep the do-files. The do-files are the Stata-specific files that keep a record of everything that has been changed. All of the instructions (clean and transform this dataset, produce this and that analysis) go in the do-file. In the do-file you say: take this raw dataset with census data, merge it with this electoral data, and then save the file that results from the merge in the data-clean folder. The do-file keeps a record of the instructions where you are telling Stata to do all of these things.
The output folder is for your tables and figures. Suppose you produce a table that shows how many elections were held in recession years versus non-recession years. You can generate that with Stata, and if you want a table in a clean format, or a figure such as a bar chart showing how many elections were held in recession versus non-recession years, you add an instruction in your do-file that says: make this table, or make this graph, and then save it in the output folder.
I suggest that you start setting up this very basic folder structure now, even if you have not yet downloaded Stata. It is perfectly fine to just go on your computer and create those folders. It is important that you start understanding why we are doing this and that you have a clear role for each of those folders. And of course, as you move on, you can adapt the folder structure as you see fit.
Recap
- Reproducibility means that others can retrace your analysis. “Others” here also includes you in the future. It is easy to assume that we are going to remember everything that we did but, as I know from experience, you will not, and most people will not remember every step of the way. You want to make it reproducible for yourself as well, so that when you pick up the same project later you know what was done and why.
- Separating the data from the code is key. It is a requirement for reproducible research: without separating the data from the code, there is no way the research can be reproducible.
- A clear folder structure, knowing exactly what types of files each folder contains, is also essential for good analysis and for reproducible research.
Why this matters
Reproducible workflows make your analysis clearer, and also more reliable and more transparent. You might make a mistake, and someone else who looks at your work will be able to see that in a particular line of code you did something slightly wrong that is compromising the rest of the analysis. With a small fix, something that would otherwise fall apart can easily be put right. That is what reproducibility allows you to do.
And trust me: knowing how to set up a clear folder structure now will save you a lot of headaches in the future. I encourage you to get started by setting up a folder structure that at least resembles the one suggested here for when you start conducting quantitative analysis.