[Excerpts taken from the article “A Journal-Based Replication of ‘Being Chosen to Lead’” by Allen Drazen, Anna Dreber, Erkut Ozbay, and Erik Snowberg, posted at Snowberg’s website at the California Institute of Technology]

“More replication seems needed, and there is also a need for it to be done quickly, before false positive (or negative) findings are able to take a prominent place in the literature. This note proposes and executes a proof-of-concept of a novel mechanism for replication: journal-based replication.”

“In this mechanism, a replication attempt is contracted by the journal after a study is accepted for publication, but (ideally) before the actual publication occurs.”

“Our own proof-of-concept occurred within the Journal of Public Economics, considered a top field journal in economics. The experimental study selected for replication, with the enthusiastic support of the authors (who became co-authors of the current manuscript), was the basis of “Does `Being Chosen to Lead’ Induce Non-selfish Behavior? Experimental Evidence on Reciprocity,” by Drazen and Ozbay (2019). That study found that elected representatives are more responsive than appointed representatives to the concerns of their constituents, all else equal.”

“There are four broad decisions we believe are of primary importance for the sustainability and usefulness of journal-based replications. They are: whether the replication should be conducted before or after the publication decision; the size and scope of the replication attempt; who should pay for it; and what should be done with the data from the replication.”

In our case, due to the exploratory nature of this replication attempt, we felt that publication decision should not be affected by the replication attempt.”

“…for a number of reasons, we decided on an “exact” replication that would make no modifications beyond correcting typos and similar errors–including to the sample size.

“We suggest three funding models going forward. First, experimenters could include in their grants a request for funds to cover a replication attempt by a journal. Second, “open” journals often charge publication fees once an article is accepted for publication. …it is not out of the question to make replication fees a standard part journal publication fees. Third, journals, or the societies that run them, could apply for grants themselves to run journal-based replication pilot programs…”

“…we anticipated the replication would be a success. In that case, we believed that the original paper would be modified to show the estimates of the main treatment effect from each of the original and replication study, and then pool the data for all subsequent analyses.”

“We anticipated there was some chance the replication would be a failure, in which case we presumed that we would make some note of it in the published paper, and then put together a short note with the rest of the results.”

“However, we neglected a third scenario which is, historically, incredibly unlikely: we might get the opposite effect from that found in the paper. This turned out to be what happened.”

“The next section describes in more detail all the things that went wrong (and right) in our attempt at journal-based replication…”

“This paper proposes, and shows a proof-of-concept of, a novel mechanism of ensuring replication: journal-based replication. By making publication decisions before a replication is conducted, it reduces the possibility of  “file-drawer” problems.”

“Comparing the model of journal-based replication to a model in which replication is achieved through other means (but is still done), the difference would be that the costs of replication would be borne by the journal, and, more likely, the authors of the original study.”

“Finally, as people wonder what the purpose of journals is in an age of open access…it seems that enforcing replication could be one such purpose.”

To read the article, click here.