[From the article “A statistical fix for the replication crisis in science” by Valen E. Johnson at https://theconversation.com/au.]
“In a trial of a new drug to cure cancer, 44 percent of 50 patients achieved remission after treatment. Without the drug, only 32 percent of previous patients did the same. The new treatment sounds promising, but is it better than the standard?”
“That question is difficult, so statisticians tend to answer a different question. They look at their results and compute something called a p-value. If the p-value is less than 0.05, the results are “statistically significant” – in other words, unlikely to be caused by just random chance.”
[From the blog post “Is Piketty’s Data Reliable?” by Alex Tabarrok at Marginal Revolution]
“When Thomas Piketty’s Capital in the Twenty-First Century first appeared many economists demurred on the theory but heaped praise on the empirical work. “Even if none of Piketty’s theories stands up,” Larry Summers argued, his “deeply grounded” and “painstaking empirical research” was “a Nobel Prize-worthy contribution”.”
“Theory is easier to evaluate than empirical work, however, and Phillip Magness and Robert Murphy were among the few authors to actually take a close look at Piketty’s data and they came to a different conclusion: ‘We find evidence of pervasive errors of historical fact, opaque methodological choices, and the cherry-picking of sources to construct favorable patterns from ambiguous data.'”
“Magness and Murphy, however, could be dismissed as economic history outsiders with an ax to grind. …The Magness and Murphy conclusions, however, have now been verified (and then some) by a respected figure in economic history, Richard Sutch.”
[From the article “When the Revolution Came for Amy Cuddy” by Susan Dominus at nytimes.com]
“As a young social psychologist, she played by the rules and won big: an influential study, a viral TED talk, a prestigious job at Harvard. Then, suddenly, the rules changed.”
Amy Cuddy became famous for her work in social psychology; in particular, a 2010 study on power poses. That research, published in the prestigious journal Psychological Science, led to prominent media exposure on CNN, Oprah magazine, and a TED talk that has become the second most popular TED talk with a viewership of over 43 million.
The “revolution” in the headline is the replication movement, and Amy Cuddy’s work became something of a poster child for bad science that could not be replicated. This article presents a compelling look of both sides of the replication debate. On the one hand, the desire for good science and the calling out of shoddy statistical practices. On the other hand, the personal costs, including professional and public humiliation, when one’s work becomes singled out — in some cases, unfairly — for criticism. It is a great read.
In a recent opinion piece for Slate, the ubiquitous Andrew Gelman took the prestigious journal Proceedings of the National Academy of Sciences (PNAS) to task for claiming that it “only publishes the highest quality scientific research.” As a result, PNAS no longer makes that claim.
Each year, the Berkeley Initiative for Transparency in the Social Sciences (BITSS) awards prizes to researchers who have made substantial contributions to improving transparency in research practices. The prizes are names after Ed Leamer (economics) and Robert Rosenthal (psychology) through their early efforts in identifying problems in scientific publishing. On October 12th, BITSS announced the recipients of the 2017 Leamer-Rosenthal Awards,
A total of eight awards were made this year. Two were made in the area of education: Daniel Lakens for, among other things, his creation of the online course “Improving Your Statistical Inferences”; and Simine Vazier, co-founder and President of the Society for the Improvement of Psychological Science. Six other awards were made to emerging researchers for promoting and demonstrating transparent practices in their research.
[From the article “Do Neuroscience Journals Accept Replications? A Survey of the Literature,” published by Andy Yeung in the September issue of Frontiers in Human Neuroscience]
“Recent reports in neuroscience, especially those concerning brain-injury and neuroimaging, have revealed low reproducibility of results within the field and urged for more replication studies. However, it is unclear if the neuroscience journals welcome or discourage the submission of reports on replication studies.”
“Of the 465 journals reviewed, 28 (6.0%) explicitly stated that they accept replications, 394 (84.7%) did not state their position on replications, 40 (8.6%) implicitly discouraged replications by emphasizing on the novelty of the manuscripts, and 3 (0.6%) explicitly stated that they reject replications.”
Replications are pivotal for the credibility of empirical economics. Evidence-based policy requires findings that are robust and reproducible. Despite this, there has been a notable absence of serious effort to establish the reliability of empirical research in economics. As Edward Leamer famously noted, “Hardly anyone takes data analysis seriously. Or perhaps more accurately, hardly anyone takes anyone else’s data analysis seriously.” This is evidenced by the fact that replication studies are rarely published in economic journals.
However, the situation may be changing. Recently, the Deutsche Forschungsgemeinschaft (DFG) released a Statement on the Replicability of Research Results in which it emphasized the importance of replication to ensure the reliability of empirical research. Accordingly, DFG is funding a new scientific journal, the “International Journal for Re-Views in Empirical Economics (IREE)”.
IREE is a joint project of Leuphana University of Lüneburg (Joachim Wagner), the German Institute for Economic Research (DIW Berlin) (Gert G. Wagner), the Institute of Labor Economics (Hilmar Schneider), and the ZBW. Nobel laureate Sir Angus Deaton (Princeton University), Jeffrey M. Wooldridge (Michigan State University), and Richard A. Easterlin (University of Southern California) are members of the advisory board of IREE.
The International Journal for Re-Views in Empirical Economics (IREE) is the first journal dedicated to the publication of replication studies based on economic micro-data. Furthermore, IREE publishes synthesizing reviews, micro-data sets and descriptions thereof, as well as articles dealing with replication methods and the development of standards for replications. Up to now, authors of replication studies, data sets and descriptions have had a hard time gaining recognition for their work via citable publications. As a result, the incentives for conducting these important kinds of work were immensely reduced. Richard A. Easterlin notes the paradox when he states: “Replication, though a thankless task, is essential for the progress of social science.”
To make replication a little less thankless, all publications in IREE are citable. Each article, data set, and computer program is assigned a DOI. In addition, data sets are stored in a permanent repository, the ZBW Journal Data Archive. This provides a platform for authors to gain credit for their replication-related research.
Up to now, publication of replication studies has often been results-dependent, with publication being more likely if the replication study refutes the original research. This induces a severe publication bias. When this happens, replication, rather than improving things, can actually further undermine the reliability of economic research. Compounding this are submission and publication fees which discourage replication research that is unlikely to get published.
IREE is committed to publishing research independent of the results of the study. Publication is based on technical and formal criteria without regard to results. To encourage open and transparent discourse, IREE is open access. There are no publication or submission fees, and the journal is committed to a speedy and efficient peer-review process.
To learn more about IREE, including how to submit replication research for publication, click here.
Dr. Martina Grunow is Managing Editor of the International Journal for Re-Views in Empirical Economics (IREE) and is an associate researcher at the Canadian Centre for Health Economics (CCHE). She can be contacted by email at firstname.lastname@example.org.
Duvendack, M., Palmer-Jones, R.W. & Reed, W.R., 2015. Replications in Economics: A Progress Report, Econ Journal Watch, 12(2): 164-191.
Leamer, Edward E., 1983. Let’s Take the Con Out of Econometrics, The American Economic Review, 73(1): 31-43.
Mueller-Langer, F., Fecher, B.,Harhoff, D. & Wagner, G. G., 2017. The Economics of Replication, IZA Discussion Papers 10533, Institute for the Study of Labor (IZA).
[From the abstract of a recent working paper by Blakeley McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer Tackett.]
“In science publishing and many areas of research, the status quo is a lexicographic decision rule in which any result is first required to have a p-value that surpasses the 0.05 threshold and only then is consideration—often scant—given to such factors as prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain. There have been recent proposals to change the p-value threshold, but instead we recommend abandoning the null hypothesis significance testing paradigm entirely, leaving p-values as just one of many pieces of information with no privileged role in scientific publication and decision making. We argue that this radical approach is both practical and sensible.”
In a recent blog at Open Philanthropy Project summarizing some of his research on the effect of incarceration on crime, David Roodman conducted, count them, 8 replications. Here is what he said: “Among three dozen studies I reviewed, I obtained or reconstructed the data and code for eight. Replication and reanalysis revealed significant methodological concerns in seven and led to major reinterpretations of four.”
[From a letter published in the September 22 issue of Science entitled “Addressing scientific integrity scientifically”]
“To introduce greater rigor into the study of research integrity and the factors that foster or discourage responsible behavior, the participants at the Fifth World Conference on Research Integrity endorsed the “Amsterdam Agenda” (1). Under this Agenda, the newly created World Conferences on Research Integrity Foundation plans to establish a registry for research on research integrity. The registry will ask researchers to describe the integrity problem they are addressing, how the problem affects research, the intervention they are introducing to address the problem, why they hypothesize that the intervention will work, how they will assess the outcome, and their plans for data sharing.”
To read more, click here (the full text is behind a paywall)