[From the blog, “The uncanny mountain: p-values between .01 and .10 are still a problem” by Julia Rohrer, posted at The 100% CI]
“Study 1: In line with our hypothesis, …, p = 0.03.”
“Study 2: As expected, … p = 0.02.”
“Study 3: Replicating Study 2, … p = 0.06.”
“Study 4: …qualified by the predicted interaction, … p = 0.01.”
“Study 5: Again, … p = 0.01.”
“Welcome to the uncanny … p-mountains, one of the most scenic accumulations of p-values between .01 and .10 in the world! Over the course of the last years, many psychologists have learned that such a distribution of p-values is troubling (see e.g. blog posts by Simine Vazire and Daniël Lakens) and statistical tools have been developed to analyze what these distributions can tell us about the underlying evidence (Uli Schimmack’s TIVA, p-curve by Simonsohn, Nelson and Simmons p-curve)—as it turns out, quite frequently, the answer is not good news.”
“However, the uncanny p-mountains can still be seen in journal articles published in 2018. And this is probably no surprise given that (1) our intuitions about something as unintuitive as frequentist statistics are often wrong and (2) many researchers have been socialized in an environment in which such p-values were considered perfectly normal, if not a sign of excellent experimental skills. So let’s keep talking about it!”
[From the article “Q&A Adam Russell: Can automated tools reliably rate research reproducibility?” by Dalmeet Singh Chawla, published at natureindex.com]
[From the blog “How Freely Should Scientists Share Their Data?” by Daniel Barron, published at blogs.scientificamerican.com]
“At the beginning of graduate school, I decided I wanted to study how epileptic seizures damage the brain. I was in something of a pickle: I wanted to use magnetic resonance imaging (MRI) to study this damage, but I didn’t have access to MRI data of patients with epilepsy. Even if I had that data, I didn’t know much about programming or mathematics or physics, so I couldn’t have created ex nihilo the software tools to analyze the data anyway. So, I was driven and energetic and wanted to study epilepsy, but I didn’t have the data or tools to work with.”
“… At around this time, I first heard about the Open Science movement—the increasingly popular belief that scientific methods and data should be freely available. The overall goal is to make science as democratic and accessible as possible. To do this, Open Scientists make their data, methods and code (computer programs that analyze data) openly available to the public. Open Scientists also share with their colleagues, which, as I discovered as a graduate student, can be a great boon to science.”
“I also heard cautionary tales that the Open Science movement had a dark side, that “openness” had, at times, devolved into bullying and theft. Some compared the Open Science movement to Communism: good in principle, impossible in practice. In informal settings—at dinner, over drinks—I was reminded that science was a competitive business.”
“But I didn’t worry that much until early this July.”
[From the abstract of the paper, “A Unified Framework to Quantify the Credibility of Scientific Findings”, by Etienne LeBel, Randy McCarthy, Brian Earp, Malte Elson, and Wolf Vanpaemel, published in the journal, Advances in Methods and Practices in Psychological Science]
[From a letter to the editor by Arthur C. Evans Jr., chief executive of the American Psychological Association]
“We are proud that psychologists are at the forefront of those calling for reassessment of earlier research. The American Psychological Association has embraced the move toward more open science, encouraging the scientists whose work we publish to deposit their data in online repositories for other researchers to scrutinize and reanalyze.”
“Recognizing that this process depends on broad awareness of data availability, we have also begun to mark articles with badges denoting that the study methods were preregistered and that the author is sharing data or related materials.”
[From the blog “A Great Day for Open Policy Analysis” by Fernando Hoces de la Guardia (BITSS Postdoc) posted at http://www.bitss.org]
“Earlier this year, I headed over to the CBO [U.S. Congressional Budget Office] to share my efforts to “reverse engineer” one of their debated reports, on the projected effects of a minimum wage increase. I recreated the CBO analysis in a fully transparent and reproducible format, using tools that are now common in the open science community.”
“Analysts at the agency seemed very receptive to the idea of OPA [Open Policy Analysis]… So you can imagine how excited I was to see the following tweet from CBO yesterday.”
[From the article “There Is More to Behavioral Economics Than Biases and Fallacies” by Koen Smets, published at the Behavioral Scientist]
“In some cases, follow-up studies have unmasked outright frauds, like Diederik Stapel, who fabricated and manipulated data to show that meat eaters are more selfish than vegetarians. Similarly, follow-up studies have exposed the dubious-but-perhaps-not-quite-fraudulent methods of people like Brian Wansink, whose papers have repeatedly been found to contain inaccuracies and errors undermining the validity of his conclusions.”
“But other “failed replications” are not quite so unequivocal once one looks behind the headlines. Human behavior is complex, and sensitive to context and circumstances. It is unwise to take an observed effect as gospel (we’ll come back to that in a moment), but it is equally unwise to take a single failed replication as proof that the originally observed effect does not exist.”
“…. We need to continue to replicate, in different circumstances, and build up the model of human cognition and behavior—much like how a sculpture is made. Start with a rough wire mesh, add some dollops of clay, stand back and observe, trim and refine. Doesn’t quite fit? Remove and start again. The difference is that a sculpture usually gets finished in the end. Behavioral economics will be a work in progress for a long time to come. But with the right sculptors, there is hope yet for a masterpiece.”
[From the blog, “The replication puzzle: why do policies work there but not here?” by Tom Graham, posted at Apolitical]
“In the US, the Nurse-Family Partnership is considered a proven program. It sends nurses to visit poor first-time mothers, and in three randomised controlled trials (RCTs) it’s been shown to have a big impact on the health and wellbeing of mother and child. The model was imported to the UK on that basis and shifted to a much bigger scale. But when it was already being implemented at 132 sites across the country, they tested it in an RCT — and found it had no effect.”
“The Nurse-Family Partnership is one of the clearest recent cases of a puzzle that bedevils any attempt to replicate a program that’s working somewhere else: generalisability. It’s the question mark at the centre of any scale up: if we replicate an approach in a new context, can we expect the impact to be similar?”
[From the blog “Psychology’s New Normal” by Stephen Lindsay, posted at the Center for Open Science’s website]
“As one means of encouraging these transparent science practices, the Center for Open Science developed the idea of awarding badges to articles that met certain criteria. The data badge is awarded if the data needed to reproduce the analyses reported in the article can be directly accessed from a permanent, third-party site by other researchers. The materials badge is awarded if the materials needed to reproduce the procedure can be directly accessed. And the preregistration badge is awarded if the researchers show that they had a detailed plan for how they would conduct and analyze the study before they looked at the data.”
“Eric Eich, my predecessor as Editor in Chief of Psychological Science, made the journal the launch vehicle for such badges, beginning in 2014. Uptake was gradual but steady, and may have been assisted by a report by Kidwell et al. (2016) with evidence that the badges were making a real difference to the likelihood that other researchers could access the data associated with a paper.”
“…This month, July of 2018, the Table of Contents for Psychological Science is like a billboard announcing the new normal. As you can see, 13 of the 15 “regular” articles received the data badge (nine also received the materials badge and we had three of the once-rare triple-badgers). We’re not done yet. The quality of the preregistrations that earn badges is still very mixed and we cannot guarantee that other researchers will be able to reproduce the analyses of articles that earn the data badge. But we are moving in the right direction. And it is very exciting to hear that other societies and other journals are also taking up badges as a way of making transparency normative.”
[From the article, “Replication Failures Highlight Biases in Ecology and Evolution Science” by Yao-Hua Law, published at http://www.the-scientist.com]
“As robust efforts fail to reproduce findings of influential zebra finch studies from the 1980s, scientists discuss ways to reduce bias in such research.”
“…Available data suggest that questionable research practices are common enough in ecology and evolution research to warrant concern. Last month, Hannah Fraser, an ecologist at University of Melbourne, and colleagues surveyed more than 800 ecologists and evolutionary biologists and found that many of the researchers—mostly midcareer and senior—admitted to at least one instance of selective reporting (64 percent), use of the flexible stopping rule (42 percent), or having changed hypotheses to fit their results (51 percent).”
The Replication Network 
You must be logged in to post a comment.