GOODMAN: Hold the Bus!

A recent news piece in Nature reported in glowing terms on the “first analysis of ‘pre-registered’ studies”, stating that “[pre-registration] seems to work as intended: to reduce publication bias for positive results.” There are reasons to be somewhat dubious about this claim.
The analysis in question appears in a preprint, “Open Science challenges, benefits and tips in early career and beyond”. The analysis is a small part of the paper, occupying about half a page of an 11-page document. The paper draws no strong claims from the data; the Nature story goes well beyond what the paper says, though I can easily believe the authors waxed proudly about their results when interviewed by the journalist.
The preprint is really an essay on preregistration and true to its title discusses challenges and risks, esp. for early stage investigators, as well as potential benefits. The authors are proponents of preregistration and reach the expected conclusion that the benefits outweigh the risks.
The angle of the Nature story is that preregistration cuts publication bias by increasing the proportion of null results that are published. The reporter drives the point home with a graphic (reproduced below) that vividly shows the increase in null findings in bright red: 55-66% with pre-registration vs. 5-20% without. Sounds convincing.

trn(goodman, 20190119)

But hold the bus! There are several problems.
1. The data comes from a biased sample. The first wave of pre-registrants are presumably people committed to the ideal who want to do it right. There is no rigorous way to extrapolate from this (or any) biased sample to the population as a whole.
2. The study confounds two factors: preregistration and journals’ willingness to publish null results. There’s no way to allocate the treatment effect without a study that separates the factors. Perhaps the results would be just as good if journals were eager to publish null findings that weren’t preregistered.
3. The output variable is, at best, a surrogate for what we actually want. Is our goal really to increase the number of null results in the literature? If so, there are many trivial ways to accomplish this. No. I suspect the true goal is to improve the quality of science. The proportion of nulls is somehow thought to be an indicator of quality, although I’m not aware of any evidence to support this claim.
The research enterprise is a complex dynamic system driven by economic forces. Before mucking about with something as central as the criteria for publication, one needs to consider long term unintended consequences. Unless we repeal publish-or-perish, everyone in the field will continue publishing in order to keep their jobs. Unless we improve the quality of people in the field or give them more money or time to do research, we’ll be stuck with the same researchers publishing the best papers they can with limited resources. 
Preregistration will make it harder and more costly to do research and the likely first-order effect will be to reduce the amount of research. If the change preferentially makes it harder to do good research, the outcome will be a classic unintended consequence: we will worsen, not improve, overall research quality.
Are you willing to risk these unintended consequences without a proper controlled study of the proposed change? I hope not. 
This begs the question of how to do such a study as there is no obvious way to blind participants as to whether they’re in the treatment (preregistration) or control group. An alternative is to look at the experience in other fields where preregistration has been in effect for a long time, for example, medical research. To date, this has not been subject to rigorous study. However, it is this author’s assessment that the outcome is not as positive as the Nature report suggests.
Nat Goodman is a retired computer scientist living in Seattle Washington. His working years were split between mainstream CS and bioinformatics and orthogonally between academia and industry. As a retiree, he’s working on whatever interests him, stopping from time-to-time to write papers and posts on aspects that might interest others. He can be contacted at natg@shore.net.

2 Comments on “GOODMAN: Hold the Bus!

  1. “This begs the question of how to do such a study as there is no obvious way to blind participants as to whether they’re in the treatment (preregistration) or control group.” I think the best method will be to randomize at the point of “revise and resubmit”: Half of such decisions would also be granted an in-principle acceptance after 1 additional round of review: “If you agree to satisfy reviewer comments and collect one more round of data, results will published regardless of outcome.” Other half will have a typical “please collect one more round of data.” Major benefit of this design is that it does not rely on author self-selection to submit a Registered Report. Two limitation of it: 1, this only applies to disciplines where such Revise and Resubmits are common and 2, not blinded.

    “Preregistration will make it harder and more costly to do research and the likely first-order effect will be to reduce the amount of research.” I don’t follow how this is true. I suspect that creating a preregistration takes more time up front, which is likely/possibly saved on the backend. This *might* apply to Registered Reports, where journals and reviewers are likely to put up sample size requirements, and the stage 1 review process does slow down starting a project. But I think it’s far from a given at this point. Especially considering design improvements from stage 1 peer review and reduced time to final publication (IPA a1 1 journal instead of shopping it around to multiple), time costs could easily be a wash or an improvement.

    Like

  2. Pingback: An Interesting Thread on Pre-registration | The Replication Network

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: