[Excerpts taken from the article “Do Pre-analysis Plans Hamper Publication?” by George Ofosu and Daniel Posner, published in the AER: Papers and Proceedings]

“Pre-analysis plans (PAPs) have been criticized… that PAPs generate dull, lab-report-style papers that are disfavored by reviewers and journal editors and thus hampered in the publication process.”

“To the extent that scholars who register and adhere to PAPs are disadvantaged in publishing their papers, researchers may be disincentivized from preregistration. This risks undermining the benefits for research credibility that the broader adoption of PAPs is thought to offer.”

I. Publication Outcomes of NBER Working Papers with and without PAPs

“We analyze papers issued as NBER working papers between 2011 and 2018, the period corresponding with the rise of preregistration in the economics discipline.”

“During this time span, NBER issued 8,706 working papers, of which 973 (11 percent) were experimental and thus were plausible candidates for preregistration. Fifty-three percent of these experimental working papers were subsequently published in peer-reviewed journals, with 13 percent landing in top-five outlets.”

“To assess whether PAPs affect the likelihood of publication, we coded whether each of these papers mentioned a PAP. We then calculated the publication rates of papers with and without PAPs.”

“Papers reporting the results of studies that followed PAPs were 10 percentage points less likely to be published by December 2019 than papers that did not mention a PAP (44 percent versus 54 percent; p < 0.1). However, conditional on being published, papers with PAPs were 39 percentage points more likely to land in a top-five journal (61 percent versus 22 percent; p < 0.01).”

II. Do Studies with PAPs Generate More Citations?

“…we collected data from Google Scholar on the number of citations to the 82 experimental NBER working papers that mention a PAP and a sample of 100 of the 477 published and 100 of the 414 unpublished experimental NBER working papers that do not mention a PAP.”

“Controlling for the number of years since being issued as an NBER working paper, whether the paper was published, and whether it was published in a top-five outlet…we estimate that having a PAP is associated with 14 additional citations…This represents more than a 40 percent increase over the 32 citations achieved by the median NBER working paper in our sample.”

III. Conclusion

“In keeping with the concerns of some PAP critics, who worry that fidelity to a PAP will lead to an uninteresting, mechanical paper that will be disadvantaged in the review process, we find that papers with PAPs are in fact slightly less likely to be published. However, we also find that, conditional on being published, papers with PAPs are more likely to land in top-five journals and are more likely to be cited.”

To read the article, click here.