1. Reproduction and replication
Replication has gained increased attention because of the replication crisis (link). The idea of replication is, however, much older. To produce reliable knowledge, researchers must be able to repeat each other’s (and their own) studies and analyses. Doing the same or similar study twice or multiple times has always been regarded as providing additional support for the study’s conclusions. Replication has taken on a more precise meaning in recent years.
Often a distinction is made between reproduction and replication.[1] Reproductions are almost exact copies of the original study. Reproductions make use of the same data or sources and the same research protocol. Replications make use of the same research protocol as well but may use new data or sources. Reproductions are better suited for quality control of methods and replications better suited to see whether conclusions generalize to new populations or new data. Both are key in putting conclusions on a better epistemic footing.
On a more fine-grained level, we can distinguish between computational reproducibility and replicability: Computational reproducibility is achieved when repeating a study using the exact same methods and data yields the exact same results as the original analysis; replicability is achieved when repeating a study using the same methods but different data yields qualitatively similar results and the same conclusions as the original.
2. From replicability to reproduction
Several authors noted that many studies are not easy (or not possible) to reproduce or replicate. Some studies lack proper documentation of the research protocol or of steps taken during the study. Some reports also miss details concerning the data or how the data was collected. This all shows the need for replicability. By better documentation and providing more details, studies are made easier to replicate by other scholars.
Various proposals were made to make studies replicable. Examples are standardized research protocols, preregistrations[2], data-repositories and sharing of programming code. What is needed to make studies replicable differs by discipline.
Increased replicability aids in making studies more available for replication but also makes studies more transparent. Increased transparency makes it easier to assess the quality of studies and to note limitations. From an outside perspective, reproducibility may seem like a trivial achievement. But standard scientific practice often doesn’t guarantee reproducible results: Threats to reproducibility, such as inaccessible and poorly documented study materials, methods, analysis code and data, reliance on workflows with little protection against human (and machine) error, and lacking attempts to verify the computational reproducibility and replicability of published research, have been documented in many scholarly disciplines.
There is now a wide consensus that most, if not all disciplines, benefit from better documentation of methodology, research protocols, data used and other steps taken during research. In order to be replicable, more is needed like sharing of data-repositories and/ or programming code.
3. A replication crisis
The call for increased attention to replication was fed by the ongoing replication crisis in the social and biomedical sciences. Reperforming a number of key-studies in social psychology, bio-medicine and other disciplines yielded different results or different conclusions than the original studies. This led to increased reflection on research standards within disciplines and some doubts concerning the reliability of methods used.
Whereas replication-studies were marginal at best, the replication crisis led to an increase in replications and greater awareness for the need of replication-studies. The awareness spilled over into other disciplines like the humanities and natural sciences .
4. Institutionalized replicability
Growing awareness of the need for replicability highlighted problems in achieving this goal. More documentation in research needs a commitment from individuals scientists. Doing replicable research also requires more time and energy. In a context with high publication-pressure and limited time or resources, replicability is not a key focus of many researchers. Therefore increased replicability requires changes in research climate and support from academic institutions. Many institutions have pledged support for replicability and related concerns like open science. More efforts are, however, needed to ensure a higher ration of replicable studies and more robust research.
[1] Different categorizations and different definitions of ‘reproducibility’ are around. Most, however, draw on similar ideas like a high degree of similarity to the original study and its goal of corroborating results.
[2] Preregistrations are statements of studies to be performed. Preregistrations include details on the research protocol and the data to be collected. They are made publicly available before the study is performed. Preregistrations help in avoiding data cherry-picking or tweaking hypotheses to get significant results.