[From the article “To catch misconduct, journals are hiring research integrity czars” by Ivan Oransky and Adam Marcus, published at STAT]
“Christopher, an editorial assistant at FEBS — for Federation of European Biochemical Societies — Letters in Heidelberg, Germany, is one of a small but growing number of research integrity czars whom publishers are employing to help police their pages.”
“…Scientific journals’ creation of dedicated positions for rooting out misconduct before publication comes amid growing awareness of such issues, and stems from a recognition that spot-checking and other ad hoc arrangements were insufficient.”
“…In an editorial earlier this year, Christopher recounted the story of a dozen manuscripts she’d reviewed after they raised red flags, revealing ‘serious, systematic and large-scale fabrication of research results.'”
“‘Cases like this confirm yet again the importance and obvious benefits of screening articles before publication,’ she wrote. ‘Our constant vigilance is a service to authors and the wider scientific community.’”
To read the article, click here.
[From the working paper, “8 Easy Steps to Open Science: An Annotated Reading List” by Sophia Crüwell et al., posted at PsyArXiv Preprints]
“In this paper, we provide a comprehensive and concise introduction to open science practices and resources that can help students and researchers with no background knowledge understand and implement best practices.”
“…we curate and summarise reviews, tutorials, and metascientific research related to eight topics: Open Science, Open Access, Open Data, Preregistration, Reproducible Analyses, Best Practices in Statistics, Replications, and Teaching Open Science. For each topic, we provide a summary of one publicly available, published, peer-reviewed article and suggest additional readings. In doing so, we aim to make open science practices both understandable and actionable to readers.”
Category: NEWS & EVENTS
Tags: confidence intervals, Open access, Open data, Open Science, p-value, Preregistration, Reading list, Registered Reports, replication, Reproducibility, Teaching Open Science
[From the working paper, “Sound Inference in Complicated Research: A Multi-Strategy Approach” by Sanjay Srivastava, posted at PsyArXiv Preprints]
“Preregistration is effective because it creates decision independence: analytic decisions are the same regardless of the specific and potentially spurious features of the data being analyzed, instead of being overfit to them. But preregistration can be difficult in practice for some complicated research paradigms, including longitudinal studies, statistical modeling, machine learning, and other intensively multivariate designs and analyses where key decisions may be difficult to anticipate or make in advance.”
“When simple preregistration is not practical, other strategies that create decision independence can be used to augment it or in place of it. This manuscript discusses standardization, blind analyses, data partitioning, supporting studies, coordinated analysis, and multiverse analyses as additional strategies that can be used to create adaptive preregistrations or to handle decisions that were not anticipated.”
[From a post by Andrew Gelman at his blog, Statistical Modeling, Causal Inference, and Social Science]
“If researchers and policy makers continue to view results of impact evaluations as a black box and fail to focus on mechanisms, the movement toward evidence-based policy making will fall far short of its potential for improving people’s lives.”
“I agree with this quote from Bates and Gellenerst, and I think the whole push-a-button, take-a-pill, black-box attitude toward causal inference has been a disastrous mistake. I feel particularly bad about this, given that econometrics and statistics textbooks, including my own, have been pushing this view for decades.”
“Stepping back a bit, I agree with Vivalt that, if we want to get a sense of what policies to enact, it can be a mistake to try to be making these decisions based on the results of little experiments. There’s nothing wrong with trying to learn from demonstration studies (as here), but generally I think realism is more important than randomization. And, when effects are highly variable and measurements are noisy, you can’t learn much even from clean experiments.”
[From the blog, “Gazing into the Abyss of P-Hacking: HARKing vs. Optional Stopping” by Angelika Stefan and Felix Schönbrodt, posted at Felix Schönbrodt’s website at http://www.nicebread.de]
“Now, what does a researcher do when confronted with messy, non-significant results? According to several much-cited studies (for example John et al., 2012; Simmons et al., 2011), a common reaction is to start sampling again (and again, and again, …) in the hope that a somewhat larger sample size can boost significance. Another reaction is to wildly conduct hypothesis tests on the existing sample until at least one of them becomes significant (see for example: Simmons et al., 2011; Kerr, 1998 ). These practices, along with some others, are commonly known as p-hacking, because they are designed to drag the famous p-value right below the mark of .05 which usually indicates statistical significance. Undisputedly, p-hacking works (for a demonstration try out the p-hacker app).
“… P-Hacking exploits alpha error accumulation and fosters the publication of false positive results which is bad for science. However, we want to take a closer look at how bad it really is. In fact, some p-hacking techniques are worse than others (or, if you like the unscrupulous science villain perspective: some p-hacking techniques work better than others).”
“As a showcase, we want to introduce two researchers: The HARKer takes existing data and conducts multiple independent hypothesis tests (based on multiple uncorrelated variables in the data set) with the goal to publish the ones that become significant.”
“… the Accumulator uses optional stopping. This means that he collects data for a single research question test in a sequential manner until either statistical significance or a maximum sample size is reached. “
“… To conclude, we have shown how two p-hacking techniques work and why their application is bad for science. We found out that p-hacking techniques based on multiple testing typically end up with higher rates of false positive results than p-hacking techniques based on optional stopping, if we assume the same number of hypothesis tests.”
[From Felix Schönbrodt’s Twitter account, @nicebread303]
“The p-hacker app just UNLOCKED the most requested PRO FEATURE: Subgroup analyses!!!”
“Check if you can find the effect for, say, young males. This is soo theoretically interesting. Now you can get the p-hacking skills of the experts FOR FREE!!!”
To check out the app and have hours of p-hacking fun, click here.
[From the article, “Psychology’s Replication Crisis Is Running Out of Excuses” By Ed Yong, published in The Atlantic]
“The Many Labs 2 project was specifically designed to address these criticisms. With 15,305 participants in total, the new experiments had, on average, 60 times as many volunteers as the studies they were attempting to replicate. The researchers involved worked with the scientists behind the original studies to vet and check every detail of the experiments beforehand. And they repeated those experiments many times over, with volunteers from 36 different countries, to see if the studies would replicate in some cultures and contexts but not others.”
“…Despite the large sample sizes, and the blessings of the original teams, the team failed to replicate half of the studies they focused on.”
“…Likewise, Many Labs 2 ‘was explicitly designed to examine how much effects varied from place to place, from culture to culture,’ says Katie Corker, the chair of the Society for the Improvement of Psychological Science. ‘And here’s the surprising result: The results do not show much variability at all.’ If one of the participating teams successfully replicated a study, others did, too. If a study failed to replicate, it tended to fail everywhere.”
“It’s worth dwelling on this, because it’s a serious blow to one the most frequently cited criticisms of the ‘reproducibility crisis’ rhetoric. Surely, skeptics argue, it’s a fantasy to expect studies to replicate everywhere. ‘There’s a massive deference to the sample,’ says Nosek. ‘Your replication attempt failed? It must be because you did it in Ohio and I did it in Virginia, and people are different. But these results suggest that we can’t just wave those failures away very easily.’”
[From the article “Making economics transparent and reproducible” by Tyler Smith, published on the American Economic Association’s website]
“The AEA spoke with Miguel about the replication problem in economics and how the next generation of researchers is embracing new tools to make the profession more credible.”
…
“AEA: There’s a growing debate about the credibility of the social sciences, what some are calling a “replication crisis.” What kind of challenge do you think the social sciences and economics are facing?”
“Miguel: There’s been increasing evidence really across the social sciences, and economics is no exception, that publication bias is a real concern.”
…
“AEA: What’s the one problem you think we really need to tackle?”
“Miguel: I do think that an absolute, fundamental improvement which has already occurred to a large degree has been sharing of data and statistical code. … The concern around that is that more and more people are using proprietary data. So the trend towards open data and data sharing, which has been so strong, is sort of stalling out a little bit right now. And it’s not clear exactly how to solve that problem…”
To read the article and/or listen to a full-length audio version of the interview, click here.
Two weeks ago, on Halloween, I wrote a post about how to conduct a replication study using an approach that emphasizes which tests might be run in order to avoid the perception of a witch hunt. The post is based on my paper with Benjamin D.K. Wood, which I recently presented at the “Reproducibility and Integrity in Scientific Research” workshop at the University of Canterbury. When Ben and I first submitted the paper to Economics E-journal, we received some great referee comments (all of which are public) including requests by an anonymous referee and Andrew Chang to include in the paper a list of what not to do – a list of don’ts.
We spent some time thinking about this request. We realized that what the referees wanted was a list of statistical and econometric no-nos, especially drawing on the most controversial replication studies funded by the International Initiative for Impact Evaluation (3ie) while we were both there. However, our role at 3ie was to be a neutral third party, at least as much as possible, and we didn’t want to abandon that now.
At the same time, we did learn a lot of lessons about conducting replication research while at 3ie, and we agreed that some of those lessons would be appropriate don’ts. So we added a checklist of don’ts to the paper that was ultimately published. Here I summarize three of these don’ts. I should note that I’m talking here about internal replication studies, which is when the replication researcher uses the original data from a publication to check whether the published findings can be exactly reproduced and are robust, particularly those findings supporting conclusions and recommendations.
When conducting a replication study, don’t confuse critiques of the original research with the replication tests or findings. Certainly, critiques of the original research can motivate the choice of replication exercises, and it is fine to present critiques in that context. But often there are critiques that are separate from what can be explored with the data. For example, a replication researcher might be concerned that the fact that treatment and controls groups were unblinded may mean the published findings are biased.
This concern about the original research design may be valid, but it is not something that can be tested through replication exercises. Simply identifying this concern is not a replication finding. We saw many examples where replication researchers interspersed their critiques of the motivation or design of the original research with their replication exercises and results. Mixing these two types of analysis contributed to some of the biggest controversies that we witnessed.
Don’t conduct measurement and estimation analysis (which some call robustness testing) before conducting a pure replication. (See here and here for more on terminology and the 3ie replication program.) Often replication researchers begin a study motivated by questions of robustness and may even take for granted that a pure replication (which is applying the published analysis methods to the original data) would reproduce the published results.
While skipping the pure replication may seem like a way to save time, conducting the pure replication often has the benefit of saving time. The pure replication is the best way for the replication researcher to familiarize herself with the data, methods, and findings of the original publication, and missing a problem at the pure replication stage is only going to confuse the measurement and estimation analysis.
Even more to the point, some consider pure replication the only stage of the research that should be called “replication”, and therefore the only results that should be reported as replication results. It is important for a replication researcher to be able to make a clear statement about the results at this stage.
Don’t present, post or publish replication results without first sharing them with the original authors. Replication research is, unfortunately, often a contentious undertaking. Replication researchers are advised to take the high road and communicate with original authors about their work – ideally from the beginning, even if the data are already publicly available. We saw cases where the replication researchers made mistakes that the original authors caught, so communication can save face on both sides.
There is a real concern about the original authors scooping a replication study by posting a correction without citing the replication researchers. We have seen this happen. Some approaches to addressing it include publicly posting the replication plan in advance. This research transparency approach serves multiple purposes, but one is putting a name and timestamp on the work that might lead to corrections. Another approach is to document the dates and subjects of communications with original authors and include this information, as an acknowledgement or footnote, in the replication study.
Perhaps one of our most important don’ts is don’t label the difference between a published result and a replication study results an “error” or “mistake” without identifying the source of the error. Just because the second estimate is different than the first does not make the second right. Ben and I already blogged about this don’t recommendation here on the World Bank Development Impact blog.
Recent revelations, such as last month’s report of the retraction of 15 articles by well-known Cornell food researcher, Brian Wansink, remind us that replication research is as important to the advancement of the natural and social sciences as ever. My hope is that more researchers accept the responsibility of conducting replication research as part of their contribution to science. The advice presented in the which tests paper and summarized in my last post and this one is intended to help them get started.
Annette N. Brown, PhD is Principal Economist at FHI 360, where she leads efforts to increase and enhance evidence production and use across all sectors and regions. She previously worked at 3ie, where she directed the research transparency programs, including the replication program.
BBC Radio just produced an interesting and balanced program about the replication crisis, with a focus on psychology.
Interviewees include John Bargh, Susan Fiske, John Ioannidis, Brian Nosek, Stephen Reicher, Diederik Stapel and Simine Vazire. One of the highlights is hearing Diederik Stapel say, in his own voice, “I made up the data.”
And I like this closing from the narrator: ““Ironically, psychology may now be ahead of the other sciences in putting its house in order”.
The program is about 30 minutes long. To listen, click here.
You must be logged in to post a comment.