Andrew Gelman had a great post yesterday that highlights a major issue — a really major issue — with replication. The problem is, there is no commonly accepted definition of what a “replication” is. Even when a definition is provided, there is no commonly accepted standard for how to interpret the results of a replication.

The post consists of a series of email excerpts between the author of an original study (Dan Kahan) and the co-authors of a study that claimed “failure to replicate” his study (Christina Ballarini and Steve Sloman), with occasional commentary from Gelman.

The post goes like this:

— Kahan emails Ballarini and Sloman to dispute their claim that they “failed to replicate” his study.

— Ballarini and Sloman both agree that they should not have said their study “failed to replicate” Kahan’s.

— Kahan asks that they make an effort to publicly correct the record.

— Sloman responds by saying that he didn’t really mean that their study didn’t “fail to replicate.” He says “I stand by our report even if you didn’t like one of our verbs [replicate].” 

— Kahan then writes a paper refuting the claim that Ballarini and Sloman “failed to replicate” his research (Title of paper = “Rumors of the ‘Nonreplication’ of the ‘Motivated Numeracy Effect’ are Greatly Exaggerated”)

Kahan’s conclusion: “This is a case study in how replication can easily go off the rails. The same types of errors people make in non-replicated papers will now be used in replications.”

Alternatively, one could argue this is NOT a case study in how replication can easily go off the rails. Rather, it illustrates that there are no rails.

To read Gelman’s post in its entirety, click here.