Professor Eric-Jan “EJ” Wagenmakers, Professor of Psychology at the University of Amsterdam, has been a leading advocate for pre-registration, replication, and the use of Bayesian statistics, particularly in replication studies. An interview that highlights his professional contributions can be found here.
Readers interested in replications should check out his course, “Good Science, Bad Science”. While focused on psychological science, the course should be of interest to many social scientists, including economists.
Over 16 classes, students are walked towards the following goals:
— “To understand the current “crisis of confidence” in psychological science, and learn what can be done to fix it.”
— “To obtain guidance and practical experience in designing informative and honest replication experiments through preregistration.”
— “To submit four high-quality preregistration proposals to peer-reviewed international journals for ‘in principle acceptance'”
To learn more about the course, click here.
[This blog originally appeared at the blogsite Development Impact] About a year ago, I blogged on a paper that had tried to replicate results on 61 papers in economics and found that in 51% of the cases, they couldn’t get the same result. In the meantime, someone brought to my attention a paper that takes a wider sample and also makes us think about what “replication” is, so I thought it would be worth looking at those results.
The paper in question is by Maren Duvendack, Richard Palmer-Jones and Robert Reed and appeared last year in Econ Journal Watch (see here). The paper starts with an interesting history of replication in economics. It turns out that replication goes pretty far back. Duvendack and co. cite the introductory editorial to Econometrica, where Frisch wrote “In statistical and other numerical work presented in Econometrica the original raw data will, as a rule, be published, unless their volume is excessive. This is important to stimulate criticism, control and further studies.” That was in 1933.
Various journals have made similar affirmations of the need for replication over the years. The Journal of Human Resources put it in its policy statement in 1990 – explicitly saying that it welcomed the submission of studies that replicated studies that had appeared in the JHR in the last five years. But this is missing from the current policy, which focuses more on making data and code available with published papers. The Journal of Political Economy took a different approach, and had a “confirmations and contradictions” section from 1976-1999. These explicit publication opportunities may have declined in recent times, but there has been a sharp surge in a different path to replication – the requirement that authors submit their code and dataset for a given paper. Duvendack and co. find 27 journals that regularly publish data and code – and many of these are top journals. The only development field journal that makes this list is the World Bank Economic Review. In addition, many funders now require that, after a decent interval, the data they funded be made publicly available in its entirety.
Before we look at Duvendack and co.’s review of replication trends, it’s worth taking a short detour as to what exactly replication means. Unfortunately, as it’s used in many conversations, it’s imprecise. Michael Clemens has a very nice (and very precise) paper where he lays out a number of distinctions. In this case, precision requires some verbosity, so hang on. Clemens lays out four different types (in two groups):
Replication (both sub-types use the same sampling distribution of parameter estimates and are looking for discrepancies that come from random chance, error, or fraud):
– Verification – uses the same specification, same population and same sample
– Reproduction – uses the same specification, same population but not the same sample
Robustness (uses different sampling distribution for parameter estimates and is looking for discrepancies that come from changes in the sampling distribution – as Clemens notes they need not give identical results in expectation):
– Reanalysis – uses a different specification, the same population and not necessarily the same sample
– Extension – uses the same specification, different population and a different sample.
Duvendack and co. are using a broader definition of replication (especially when compared to the paper I blogged on last year): they’re including what Clemens calls robustness. They go out, casting a wide net to look for replication studies (they include not only Google Scholar and the Web of Science, and the Replication in Economics wiki, but also suggestions from journal editors, their own collections and a systematic search of the top 50 economics journals). This search gives them 162 published studies. The time trend is interesting, as the figure below (reproduced from their paper) shows what could be an upwards trend:
One development journal that contributed significantly to these 162 studies was the Journal of Development Studies. We can’t tell how many other development papers are in the 162 since the rest of the main contributors are more general interest journals. The Journal of Applied Econometrics (JAE) is the main overall contributor to this body of work – they clock in with 31 replications – in large part because they have a dedicated replication section which can consist of pretty short summaries.
Duvendack and co. then look at the characteristics of these replications. I am going to focus on the non-JAE, non-experimental (as in experimental economics, not field experiments) studies which number 119. About 61 percent of these studies are an “exact” replication or, to use Clemens’ taxonomy, a verification study. 55 percent of studies extend the original findings. As might be expected, the majority of the 119 studies (73 percent) find a significant difference with the original result. About 17 percent confirm the previous study and 10 percent are mixed. And 26 percent of studies have a reply by the original study authors.
As Duvendack and co. point out, we shouldn’t think of the published studies as a random sample. Which brings us to incentives. Clearly getting confirmatory studies published in major journals is going to be hard, and particularly hard for simple verification studies. Turning to results that disagree with the original study, Duvendack and co. speculate that some journals might be reluctant to publish contradictions of influential authors. I can also imagine that younger researchers may be averse to taking on this particular challenge, given that influential senior researchers may show up in their career futures. Returning to the journal side of the equation, it also doesn’t look particularly good for the journal to take down one of their own papers. On another level, for journals the citation per page count is likely to be significantly lower for a replication than for an original paper (although Duvendack and co. suggest this could be alleviated by publishing very short replication papers with the longer paper as an online appendix). Finally, the incentives for authors of the original study to make replication easy are pretty weak – if someone confirms your study it’s not really a big deal, but it’s a big deal if they don’t.
So all of these factors point towards the lower likelihood of replications. There are a couple of factors that might make replications more likely. The first is there are somewhat more communities to support this than there used to be. Beyond the wiki mentioned above, Duvendack and co. have a list inside economics (e.g. 3ie’s work), but also for other disciplines. In addition, the growth in online storage, the growth in computer processing power, plus the increasing number of journals requiring the posting of code and data lower the costs of replication dramatically. And the trend is positive, so maybe there is some social support. It will be interesting to see what the future brings.
Markus Goldstein is a development economist with experience working in Sub-Saharan Africa, East Asia, and South Asia. He is currently a Lead Economist in the Office of the Chief Economist for Africa at the World Bank.
Recently, Paul Smaldino and Richard McElreath published the results of a computer simulation where scientific research is governed by “laws” of natural selection based on publishing “success.” Their finding that perverse incentives can cause “bad science” to push out good science has received much attention (see, for example, here and here).
In a related article, “Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition”, MARC EDWARDS and SIDDHARTHA ROY, professors of civil engineering at Virginia Tech University, argue that a tipping point can be reached where competition for research funding causes unethical behaviour to drive scientific integrity — and scientists with integrity — out of research.
The article is noteworthy for a number of reasons, not the least of which is a table that aggregates a number of “Perverse Incentives in Academia”. It’s also noteworthy for this statement: “…a tipping point is possible in which the scientific enterprise itself becomes inherently corrupt and public trust is lost, risking a new dark age with devastating consequences to humanity.” To read the article, click here.
[From the article “Why is so much research dodgy? Blame the Research Excellence Framework” in The Guardian] “In the UK, the Ref [Research Excellence Framework] ranks the published works of researchers according to their originality (how novel is the research?), significance (does it have practical or commercial importance?), and rigour (is the research technically sound?). Outputs are then awarded one to four stars….The Ref completely undermines our efforts to produce a reliable body of knowledge. Why? The focus on originality and accessibility – publications exploring new areas of research using new paradigms, and avoiding testing well-established theories – is the exact opposite of what science needs to be doing to solve the troubling replication crisis. According to Ref standards, replicating an already published piece of work is simply uninteresting.” To read more, click here.
My Dear Fellow Scientists!
“If you torture the data long enough, it will confess.”
This aphorism, attributed to Ronald Coase, sometimes has been used in a dis-respective manner, as if it was wrong to do creative data analysis.
In fact, the art of creative data analysis has experienced despicable attacks over the last years. A small but annoyingly persistent group of second-stringers tries to denigrate our scientific achievements. They drag science through the mire.
These people propagate stupid method repetitions (also know as “direct replications”); and what was once one of the supreme disciplines of scientific investigation – a creative data analysis of a data set – has been crippled to conducting an empty-headed step-by-step pre-registered analysis plan. (Come on: If I lay out the full analysis plan in a pre-registration, even an undergrad student can do the final analysis, right? Is that really the high-level scientific work we were trained for so hard?).
They broadcast in an annoying frequency that p-hacking leads to more significant results, and that researchers who use p-hacking have higher chances of getting things published.
What are the consequence of these findings? The answer is clear. Everybody should be equipped with these powerful tools of research enhancement!
The Art of Creative Data Analysis
Some researchers describe a performance-oriented data analysis as “data-dependent analysis”. We go one step further, and call this technique data-optimal analysis (DOA), as our goal is to produce the optimal, most significant outcome from a data set.
I developed an online app that allows to practice creative data analysis and how to polish your p-values. It’s primarily aimed at young researchers who do not have our level of expertise yet, but I guess even old hands might learn one or two new tricks! It’s called “The p-hacker” (please note that ‘hacker’ is meant in a very positive way here. You should think of the cool hackers who fight for world peace). You can use the app in teaching, or to practice p-hacking yourself.
Please test the app, and give me feedback! You can also send it to colleagues: http://shinyapps.org/apps/p-hacker
The full R code for this Shiny app is on Github.
Train Your p-Hacking Skills: Introducing the p-Hacker App
Here’s a quick walk-through of the app. Please see also the quick manual at the top of the app for more details.
First, you have to run an initial study in the “New study” tab:
When you ran your first study, inspect the results in the middle pane. Let’s take a look at our results, which are quite promising:
After exclusion of this obvious outlier, your first study is already a success! Click on “Save” next to your significant result to save the study to your study stack on the right panel:
Sometimes outlier exclusion is not enough to improve your result.
Now comes the magic. Click on the “Now: p-hack!” tab – this gives you all the great tools to improve your current study. Here you can fully utilize your data analytic skills and creativity.
In the following example, we could not get a significant result by outlier exclusion alone. But after adding 10 participants (in two batches of 5), controlling for age and gender, and focusing on the variable that worked best – voilà!
Do you see how easy it is to craft a significant study?
Now it is important to show even more productivity: Go for the next conceptual replication (i.e., go back to Step 1 and collect a new sample, with a new manipulation and a new DV). Whenever your study reached significance, click on the Save button next to each DV and the study is saved to your stack, awaiting some additional conceptual replications that show the robustness of the effect.
Many journals require multiple studies. Four to six studies should make a compelling case for your subtile, counterintuitive, and shocking effects:
Honor to whom honor is due: Find the best outlet for your achievements!
My friends, let’s stand together and Make Science Great Again! I really hope that the p-hacker app can play its part in bringing science back to its old days of glory.
(A quick side note: Some so-called „data-detectives“ developed several methods for detecting p-hacking. This is a cowardly attack on creative data analyses. If you want to take a look at these detection tools, check out the p-checker app. You can also transfer your p-hacked results to the p-checker app with a single click on the button ‘Send to p-checker’ below your study stack on the right side).
Best regards,
Ned Bicare, PhD
Providing access to the data is a prerequisite for replication of empirical analysis. Unfortunately, this access is not always granted to everyone (see here, and here). There is evidence that some of this may be due to concerns about requestors’ qualifications (see here).
In two recent papers, we investigated how willingness to share depended on the identity of the requestor. In a paper entitled, “(Un)available upon request: field experiment on researchers’ willingness to share supplementary materials”, Ernesto Reuben and I found that only 44% of economists were willing to share supplementary research materials of a published study they had promised to send “upon request”. This number was slightly higher if the request came from a high prestigious university rather than one that was less prestigious.
In “Gender, beauty and support networks in academia: evidence from a field experiment (Study 1)”, Magdalena Smyk and I asked experimental economists to share data they used in their published paper. The focus in this experiment was the role of gender: half the requests were sent by a female student, the other half by a male. Overall compliance rate was 34.7%, with no gender effect.
The fact that requestor’s identity matters little is good news of course, as it suggests a roughly level playing field. However, while compliance rates in economics are higher than many other disciplines (see here and here and here), there is still much room for improvement.
Michal Krawczyk is an Assistant Professor of Economic Sciences at the University of Warsaw, Poland.
REFERENCES
Krawczyk, M., & Reuben, E. (2012). (Un) Available upon Request: Field Experiment on Researchers’ Willingness to Share Supplementary Materials.Accountability in research, 19(3), 175-186.
Krawczyk, M., & Smyk, M. (2015). Gender, beauty and support networks in academia: evidence from a field experiment (No. 2015-43).
Recently, another sensational study from social psychology came under renewed criticism. The study, “Power Posing: Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance” , published in Psychological Science in 2010 by Dana Carney, Amy Cuddy, and Andy Yap claimed that adopting a “power pose” caused biological and psychological changes that made a person “more powerful”, in both fact and feeling. What makes this case particularly interesting is that this time, the dissenting opinion came from none other than the lead author of the study, Dana Carney. How Carney came around to change her mind about the reliability of her findings is quite interesting, and dare we say, hopeful? Replication played a role, as did increased awareness of good research practices. To read more, click here.
In a recent blog at Statistical Modeling, Causal Inference, and Social Science, ANDREW GELMAN asks the question: “Why is so much of the discussion about psychology research? Why not economics, which is more controversial and gets more space in the news media? Or medicine, which has higher stakes and a regular flow of well-publicized scandals?” He then goes on to give 5 reasons. (1) Psychology has a history of treating measurement issues in a more sophisticated manner than many other fields. (2) Psychology has been particularly prone to false confidence in experimental design. (3) Psychology is more open than other disciplines. In particular, Gelman writes that psychology is more open than economics, “which is notoriously resistant to ideas coming from other fields.” (4) The replication crisis has touched the work of top-tier academics. (5) Psychology attracts a lot of popular attention and dazzling claims are not immediately challenged by researchers on the other side of the ideological divide, as they are in economics. He concludes by saying, “The strengths and weaknesses of the field of research psychology seemed to have combined to (a) encourage the publication and dissemination of lots of low-quality, unreplicable research, while (b) creating the conditions for this problem to be recognized, exposed, and discussed openly.” To read more, click here.
The problems of publication misconduct – manipulation, fabrication and plagiarism – and other dodgy practices such as salami-style publications are attracting increasing attention. In the newly published paper “Misconduct, Marginality, and Editorial Practices in Management, Business, and Economics” (full text available here), we present findings on these problems in MBE-journals and the diffusion of editorial practices to combat them (Karabag and Berggren, 2016).
The data were collected by bibliometric searches of retracted papers in the seven major databases that cover almost all MBE journals and from two surveys to editors of these journals; the first to 60 journals with at least one public retraction, then to all journals investigated in the bibliometric study. A total of 298 journals editors answered the second survey.
The bibliometric study identified a strongly increasing trend of retraction, from 1 retraction in 2005 to 60 in 2015. Compared to the number of published papers, the figure is still very small, but since there are strong disincentives for editors to engage in retractions, the reported number is only the tip of the misconduct iceberg. As for the problem of marginality, more than half of the editors reported experiences of salami publications, which can be defined as the slicing of output into the smallest publishable unit.
The survey also enquired about specific practices to deal with these problems. We found that 42% of the journals have started to use software to detect possible plagiarism, while 30% ask authors to provide a data file. Only 6% ask the author(s) to provide information on each author’s specific roles. Many editors stated that they rely on the reviewers to detect and prevent misconduct. Despite this importance of the reviewers, less than 40% of the journals have public rewards for good reviewers, and less than half add good reviewers to the advisory board.
Only 10% of the journals publish replication studies, and the exact meaning of these answers may be doubted, in view of other studies which show a much lower percentage publishing replications. According to Duvendack et al. 2015, for example, only 3% of the studied journals actually do publish replication papers.
The discrepancy between the findings may be explained by social desirability. Editors might think or believe that it is socially desirable to publish replication studies and/or they may plan to publish them. Our survey had a free space for comments, but the editors did not comment on replication studies. Our interpretation is that editors are still not particularly interested in replication studies, even though many affirm the theoretical importance of replication for building theory and to disclose manipulation.
The paper also presents positive ideas to combat marginality by means of supporting more creative contributions. 98 journal editors provided a rich menu of ideas. We classified them into 14 sub-themes, grouped under four major themes. The first major theme, “editorial vision and engaged board”, included suggestions that editors should open up the journal and take risks, be visionary, have engaged editorial board members and change the editorial teams regularly. The second theme, “curate papers and connect authors,” comprised three sub-themes: curating and developing manuscripts, connecting authors, and constructive screening. The third major theme, labeled “open up the discussion” contained suggestions such as publish criticism instead of rejecting the papers, invite comments and involve industry specialists. The fourth theme, “go beyond the mainstream,” involved ideas on mixing disciplines, limiting individual authors and avoiding perfection.
Overall the findings indicate a broad editorial engagement with misconduct and marginality; however, we remain concerned with the amount of undetected misconduct. Several authors have argued that there is a knowledge gap in MBE due to the small number of replications and papers testing previously presented models (c.f. Duvendack et al., 2015; Kacmar and Whitfield, 2000).
Two recent cases show how valuable replication studies are. In 2010, Reinhart and Rogoff, based on a comparative sample of countries, alleged that public debt beyond a very specific level would stifle growth during periods of crisis. This conclusion was widely cited both within and outside the academic community, but when a graduate student at the University of Massachusett Amherst tried to replicate the study, he uncovered serious data flaws in the original paper which undermined both its conclusions and the theoretical assumption. The critical study was published (Herndon et al., 2014), but not in the American Economic Review, which had carried the original paper.
In the management field, Lepore (2014) re-studied the famous disk drive industry cases in Christensen´s (1997) theory on disruptive innovation. Extending the time period, the re-study found a very different pattern to the one suggested by Christensen. This finding calls for a re-assessment of the theoretical framework of disruptive innovation (Bergek et al., 2013).
The crucial point here is that none of these important and critical replication papers were published in the journals which published the original papers, and nor did these journals actively encourage submission of other replication studies. Public retractions of published papers will always be a means of last resort for a journal. Therefore it is so important to encourage close reading, reflection and replications!
REFERENCES
Bergek, A., Berggren, C., Magnusson, T. and Hobday, M., 2013. Technological discontinuities and the challenge for incumbent firms: Destruction, disruption or creative accumulation? Research Policy, 42(6), pp.1210-1224.
Christensen C. M. 1997. The innovator’s dilemma: When new technologies cause great firms to fail, Boston: Harvard Business School Press.
Duvendack, M., Palmer-Jones, R.W. and Reed, W.R., 2015. Replications in Economics: A progress report. Econ Journal Watch, 12(2), pp.164-191.
Herndon, T., Ash, M. and Pollin, R., 2014. Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38,, pp. 257-279.
Kacmar, K.M. and Whitfield, J.M., 2000. An additional rating method for journal articles in the field of management. Organizational Research Methods, 3(4), pp.392-406.
Karabag, S. F., & Berggren, C. 2016. Misconduct, Marginality and Editorial Practices in Management, Business and Economics Journals. PloS One, 11(7), pp. 1-25, e0159492. DOI: http://dx.doi.org/10.1371/journal.pone.0159492
Reinhart, C. M., and Rogoff, K.S. 2010. Growth in a Time of Debt. American Economic Review, 100, pp. 573-578.
ANDREW GELMAN has a new blog in which he takes on SUSAN FISKE and her labelling of (some) social media critics of psychological research as “methodological terrorists.” The blog is a must read for at least two reasons. First, Gelman gives a fairly detailed history of key papers and events that have led to the current loss of confidence in published research. Second, Fiske’s article makes it clear that there is a methodological battle going on in psychology (as well as elsewhere). While she doesn’t target anyone in particular, by implication her shots fall on a large division of researchers. What Fiske never acknowledges is that the reason social media has played such an important part in this struggle is because the traditional venue of academic discourse — conferences and journals — had been largely shut out to critics. Social media has allowed the critics of the status quo to achieve impressive gains. Fiske attempts to repulse this offensive. Gelman makes it clear that the critics will not fall away upon taking fire. The battle will be joined. To read more, click here.
You must be logged in to post a comment.