I am a member of the Peer Reviewers Openness (PRO) Initiative. The Pro Initiative is based on the idea that reviewers have the power to get journals to become more transparent. In particular, they encourage reviewers to request data and code from the journal when they are asked to review a manuscript. Here is the statement from their homepage:

“We believe that openness and transparency are core values of science. … The promise of open research can finally be realized, but this will require a cultural change in science. The power to create that change lies in the peer-review process.”

“We suggest that beginning January 1, 2017, reviewers make open practices a pre-condition for more comprehensive review. This is already in reviewers’ power; to drive the change, all that is needed is for reviewers to collectively agree that the time for change has come.”

This can work! I was recently asked to review a manuscript for a journal that does not require authors to provide their data and code along with their manuscript at the time of submission. In other words, a typical journal. Here is what I wrote the editor:

Dear Professor XXX,

Thank you for the invitation to review a manuscript for XXX.

I am happy to do that conditional on the authors providing their data and code so I can double check their analysis.

I am a member of the Peer Reviewers’ Openness Initiative. I also co-founded and manage The Replication Network. The bottom line is that I believe that many if not most of the research findings in empirical economics are not reliable. The only way to address this problem that I can see is to have researchers provide their data and code upon submission so that reviewers can do a satisfactory job of assessing the research.

As somebody who also sits on the other side of the desk, I know how difficult it can be to secure reviewers. I don’t want to make your job more difficult than it already is. But I do think our discipline has a serious problem and I don’t know of any other way to fix it but to encourage journals to require authors to provide their data and code.

I look forward to hearing your response.

Sincerely,

Bob Reed

Frankly, I did not expect to receive a positive response from the journal, but I am apparently a man of little faith. A few days later I received the following response:

Dear Prof. Reed,

I have contacted the author and got the following reply:

“No problem. We value transparency. The data is stored on Harvard Dataverse under the link XXX (I have attached relevant information to our application, and I attach the data itself to the message for convenience). As for the code (also attached) it is designed for the R environment.

I hope this helps in assessing our findings.”

I hope that you will now be able to accept this invitation to review:

Thanks in advance.

Best regards,

XXX

As anybody knows who ever has tried to find reviewers, good reviewers are scarce. Anything that is scarce has value. And value translates to leverage. Reviewers have the leverage to get journals to become more transparent. So…why not give it a go the next time you are asked to review a manuscript?

Bob Reed is a professor of economics at the University of Canterbury in New Zealand. He is also co-organizer of the blogsite The Replication Network. He can be contacted at bob.reed@canterbury.ac.nz.