[From the preprint, “What is Replication?” by Brian Nosek and Tim Errington, posted at MetaArXiv Preprints]

“According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs…This definition of replication is intuitive, easy to apply, and incorrect.”

“The problem is this definition’s emphasis on repetition of the technical methods–the procedure, protocol, or manipulated and measured events…If replication requires repeating the manipulated or measured events of the study, then it is not possible to conduct replications in observational research or research on past events.”

“We propose an alternative definition for replication that is more inclusive of all research, and more relevant for the role of replication in advancing knowledge. Replication is a study for which any outcome would be considered diagnostic evidence about a prior claim.”

“To be a replication two things must be true: outcomes consistent with a prior claim would increase confidence in the claim, and outcomes inconsistent with a prior claim would decrease confidence in the claim.”

“Replication is not about the procedures per se, but using similar procedures reduces uncertainty in the universe of possible units, treatments, outcomes, and settings that could be important for the claim. Because there is no exact replication, every replication is a test of generalizability.”

“This exposes an inevitable ambiguity in failures to replicate. Was the original evidence a false positive, the replication a false negative, or does the replication identify a boundary condition of the claim? We can never know for certain that earlier evidence was a false positive. It is always possible that it was “real” and we cannot identify or recreate the conditions necessary to replicate successfully.”

“But, that does not mean that all claims are true and science cannot be self-correcting. Persistent failures to replicate will narrow the universe of conditions to which the claim applies. That process may result in a much narrower, but precise set of circumstances in which evidence for the claim is replicable, or it may result in failure to ever establish conditions for replicability and relegate the claim to irrelevance.”

“Replication is characterized as the boring, rote, clean-up work of science. This misperception makes funders reluctant to fund it, journals reluctant to publish it, and institutions reluctant to reward it.”

“Replication is a central part of the iterative maturing cycle of description, prediction, and explanation. A shift in attitude that includes replication in funding, publication, and career opportunity will accelerate research progress.”

To read the preprint, click here.