[From the blog “Pre-results Review at the Journal of Development Economics: Lessons learned so far” by Andrew Foster, Dean Karlan, Edward Miguel and Aleksandar Bogdanoski, posted at Development Impact at World Bank Blogs]

“In March 2018, the Journal of Development Economics (JDE) began piloting Pre-results Review track (also referred to as “registered reports” in other disciplines) in collaboration with the Berkeley Initiative for Transparency in the Social Sciences (BITSS).”

“The motivation behind this exercise was simple: both science and policy should reward projects that ask important questions and employ sound methodology, irrespective of what the results happen to be.”

“A little over one year in, we reflect on the experience so far and offer our perspectives on the future of pre-results review at the JDE and in the discipline as a whole.”

“What we’ve learned so far”

“Referees (and editors) have found it hard on several occasions to judge submissions … not knowing the final results…”

“Work should be submitted relatively early for two reasons: (a) to maximize the possibility that feedback can be used in shaping final data collection, and (b) to ensure enough time for an R&R round and resubmission before the final data come in. This means at a minimum of three months, but ideally six months to a year, before data collection of the key outcome data.”

“Many authors have found Pre-results Review helpful in improving the ultimate quality of their research papers.”

“Reasons for rejection in Pre-results Review are similar to those in the regular review process.”

“We expected that the existence of a pre-results review submission track would eventually incentivize researchers to take on projects that they perceive as risky …our initial impression is that the Stage 1 submissions to the JDE so far do not appear to be substantially more (or less) risky than papers submitted through the normal journal review track.”

“Stage 2 of review at the JDE will likely allow for some flexibility in interpreting deviations from research designs accepted at Stage 1. This flexibility may be at odds with emerging best practices in other disciplines.”

“…because field experiments study human interactions in a real-world context, precautions need to be taken to prevent biasing participants’ behavior. …This means that we may not always be able to fully publicize accepted Stage 1 proposals until fieldwork is completed, even though this may be at odds with what is recommended in other disciplines…”

“…since development economics experiments can literally take years before data collection is complete, it may also be a while before the JDE is able to publish the first paper in the pre-results review track.”

To read more, click here.