[From the article “Pre-results review at the Journal of Development Economics: Taking transparency in the discipline to the next level” by Aleksandar Bogdanoski and Keesler Welch, published at the blogsite of the Abdul Latif Jameel Poverty Action Lab (J-PAL)]
“J-PAL interviewed JDE Editor in Chief Andrew Foster, Co-editor Dean Karlan, and BITSS Faculty Director Edward Miguel, to get some early insights from the pilot thus far.”
[From the article, “The Landscape of Open Data Policies” by David Mellor, published at the Center for Open Science blogsite]
“TOP [Transparency and Openness Promotion] includes eight policies for publishers or funders to use to increase transparency. They include data transparency, materials and code transparency, design and citation standards, preregistration, and replication policies. For simplicity, this post will focus just on the Data Transparency policy. TOP can be implemented in one of three levels of increasing rigor:”
– “Level 1, Disclosure. Articles must state whether or not data underlying reported results are available and, if so, how to access them.”
– “Level 2, Mandate. Article must share data underlying reported results in a trusted repository. If data cannot be shared for ethical or legal constraints, authors must state this and provide as much data as can be reasonably shared.”
– “Level 3, Verify that shared data are reproducible. Shared data must be made available to a third party to verify that they can be used to replicate findings reported in the article.”
“Elsevier, Springer Nature, Taylor & Francis, and Wiley have all recently adopted tiered data sharing policies that make improvements in supporting transparency easier for the journals that they publish. Each shares some characteristics and rely on similar tiers: from encouragement to share data to increasingly strong mandates to do so.”
[From the blog “A manifesto for reproducible science” by Marcus Munafò, published at wonke.com]
“…several initiatives exist that can support universities keen to improve their research quality and culture. The Forum for Responsible Research Metrics promotes the better use of metrics, for example in recruitment and promotion procedures, and in 2018 hosted “The turning tide: a new culture of research metrics”. This highlighted that more effort is needed to embed relevant principles in institutions and foster more sensitive management frameworks. And in mid-September the University of Bristol will host a workshop for key stakeholders (funders, publishers, journals) to discuss the formation of a network of universities that will coordinate and share training and best practice.”
“This network will be an academic-led initiative, ensuring not only that efforts to improve research quality are coordinated, but also that they are evaluated (given that well-intentioned efforts may lead to unintended consequences), and designed with the needs of the academic research community in mind. It will allow the myriad of new initiatives across the UK to be brought together, and new initiatives to be developed. The current focus on the quality of scientific research is best understood not as a crisis but as an exciting opportunity to innovate and modernise. UK science has a well-deserved reputation for being world-leading, but to remain world leading we will need to embrace this opportunity.”
[From the blog “The “80% power” lie” posted by Andrew Gelman in December 2017 at Statistical Modeling, Causal Inference, and Social Science]
“Suppose we really were running studies with 80% power. In that case, the expected z-score is 2.8, and 95% of the time we’d see z-scores between 0.8 and 4.8. Let’s open up the R:”
“> 2*pnorm(-0.8) [1] 0.42”
“> 2*pnorm(-4.8) [1] 1.6e-06”
“So we should expect to routinely see p-values ranging from 0.42 to . . . ummmm, 0.0000016. And those would be clean, pre-registered p-values, no funny business, no researcher degrees of freedom, no forking paths.”
“Let’s explore further . . . the 75th percentile of the normal distribution is 0.67, so if we’re really running studies with 80% power, then one-quarter of the time we’d see z-scores above 2.8 + 0.67 = 3.47.”
“> 2*pnorm(-3.47) [1] 0.00052″
“Dayum. We’d expect to see clean, un-hacked p-values less than 0.0005, at least a quarter of the time, if we were running studies with minimum 80% power, as we routinely claim we’re doing, if we ever want any of that sweet, sweet NIH funding. And, yes, that’s 0.0005, not 0.005. There’s a bunch of zeroes there.”
“And, no, this ain’t happening. We don’t have 80% power. Heck, we’re lucky if we have 6% power.”
[From the working paper, “Methods Matter: P-Hacking and Causal Inference in Economics” by Abel Brodeur, Nikolai Cook, and Anthony Heyes]
“…Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and p-hacking is a substantial problem in research employing DID and (in particular) IV methods. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.”
[From the paper, “Practical Tools and Strategies for Researchers to Increase Replicability” by Michele Nuijten, forthcoming in Developmental Medicine & Child Neurology]
“Several large-scale problems are affecting the validity and reproducibility of scientific research. … Many of the suggested solutions are systemic, focused on top-down implementation, or on the training of students, but many other solutions are practical tools and strategies that researchers can immediately implement in their own workflow. Researchers can use online tools to double check reported statistics, follow online courses to improve their statistical inference, or use different statistical frameworks. They can also start to preregister their research plans, engage in multi-lab collaborations, and increase their level of transparency and openness.”
To read the full article, click here.
[From an editorial published in Nature entitled, “Referees should exercise their rights”]
“At Nature, we recognize that our peer reviewers have certain ‘rights’. One of the most well known is the right to anonymity. Less widely known is that referees have the right to view the data and code that underlie a work if it would help in the evaluation, even if these have not been provided with the submission. Yet few referees exercise this right. They should do so.”
[From the abstract of the article, “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results”, published by Silberzahn et al. in Advances in Methods and Practices in Psychological Science]
[From the abstract of the article “Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015″, published in Nature Human Behaviour by Colin Camerer et al.]
“Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.”
To read the article, click here.
The study has generated much buzz in the media:
Buzzfeed: “In the last four years, 125 journals, mostly in the behavioral sciences, have adopted “registered reports,” …Similarly, more than 20,000 studies have been preregistered on the Center for Open Science’s website.”
Science News: “‘Replication crisis’ spurs reforms in how science studies are done”.
The Atlantic: “Fortunately, there are signs of progress. The number of pre-registered experiments—in which researchers lay out all their plans beforehand to obviate the possibility of p-hacking—has been doubling every year since 2012.”
NPR: “‘The social-behavioral sciences are in the midst of a reformation’… Scientists are..announcing in advance the hypothesis they are testing; they are making their data and computer code available so their peers can evaluate and check their results.”
Wired: “Thousands of researchers now pre-register their methodology and hypothesis before publication, to head off concerns that they’ll massage data after the fact.”
Washington Post: “the experiments scrutinized in this latest effort were published prior to a decision several years ago by Science, Nature and other journals to adopt new guidelines designed to increase reproducibility, in part by greater sharing of data”.
[From the blog, “Open science is now the only way forward for psychology” by Chris Chambers and Pete Etchells, posted in the Science Blog Network at http://www.theguardian.com]
“When we launched Head Quarters five years ago, psychology was in a pretty dark place. The field was still reeling from the impact of the Diederik Stapel fraud case – the largest perpetrated in psychology and one of the greatest ever uncovered in science. At the same time, a cascade of failures to replicate major findings was just beginning, and as if to add insult to injury, one of psychology’s most prestigious journals published a study claiming to confirm, of all things, the existence of psychic powers.”
“Psychologists were faced with one inescapable conclusion: that the research culture in the field was fundamentally flawed and needed urgent attention. Five years later, the field has taken some big steps forward toward righting the ship. Let’s take a look at some of those improvements.”
You must be logged in to post a comment.