[Excerpts taken from the preprint, “Preregistration is redundant, at best” by Aba Szollosi et al., posted at PsyArXiv Preprints]
“The key implication argued by proponents of preregistration is that it improves the diagnosticity of statistical tests…In the strong version of this argument, preregistration does this by solving statistical problems, such as family-wise error rates. In the weak version, it nudges people to think more deeply about their theories, methods, and analyses.”
“We argue against both: the diagnosticity of statistical tests depend entirely on how well statistical models map onto underlying theories, and so improving statistical techniques does little to improve theories when the mapping is weak. There is also little reason to expect that preregistration will spontaneously help researchers to develop better theories.”
“Solving statistical problems with preregistration does not compensate for weak theory. Imagine making a random prediction regarding the outcome of an experiment. Should we observe the predicted outcome, we would not regard this “theory” as useful for making subsequent predictions. Why should we regard it as better if it was preregistered?”
“On the other hand…there is nothing inherently problematic about post-hoc scientific inference when theories are strong. The crucial difference is that strong scientific inference requires that post-hoc explanations are tested just as rigorously as ones generated before an experiment — for example, by a collection of post-hoc tests that evaluate the many regularities implied by a novel theory…There is no reason not to take such post-hoc theories seriously just because they were thought of after or were not preregistered before an experiment was conducted.”
“Although preregistration does not require the improvement of theories, many argue that it at least nudges researchers to think more deeply about how to improve their theories. Though this might sometimes be so, there is no clear explanation for why we should expect it to happen.”
“One possible explanation is that researchers are motivated to improve their theories should they encounter problems when preregistering a study or when preregistered predictions are not observed. The problem with this line of argument is that any improvement depends upon a good understanding of how to improve a theory, and preregistration provides no such understanding.”
“Taking preregistration as a measure of scientific excellence can be harmful, because bad theories, methods, and analyses can also be preregistered. Requiring or rewarding the act of preregistration is not worthwhile when its presumed benefits can be achieved without it just as well.”
“…statistical methods are just tools to test implications derived from theory. Therefore, such statistical problems become irrelevant because theories, not random selection, dictate what comparisons are necessary.”
“Issues that prevent criticism of theory, such as poor operationalization, imprecise measurement, and weak connection between theory and statistical methods, need our attention instead of problems with statistical inference.”
To read the article, click here.
[Excerpts taken from the article “A Journal-Based Replication of ‘Being Chosen to Lead’” by Allen Drazen, Anna Dreber, Erkut Ozbay, and Erik Snowberg, posted at Snowberg’s website at the California Institute of Technology]
“More replication seems needed, and there is also a need for it to be done quickly, before false positive (or negative) findings are able to take a prominent place in the literature. This note proposes and executes a proof-of-concept of a novel mechanism for replication: journal-based replication.”
“In this mechanism, a replication attempt is contracted by the journal after a study is accepted for publication, but (ideally) before the actual publication occurs.”
“Our own proof-of-concept occurred within the Journal of Public Economics, considered a top field journal in economics. The experimental study selected for replication, with the enthusiastic support of the authors (who became co-authors of the current manuscript), was the basis of “Does `Being Chosen to Lead’ Induce Non-selfish Behavior? Experimental Evidence on Reciprocity,” by Drazen and Ozbay (2019). That study found that elected representatives are more responsive than appointed representatives to the concerns of their constituents, all else equal.”
“There are four broad decisions we believe are of primary importance for the sustainability and usefulness of journal-based replications. They are: whether the replication should be conducted before or after the publication decision; the size and scope of the replication attempt; who should pay for it; and what should be done with the data from the replication.”
In our case, due to the exploratory nature of this replication attempt, we felt that publication decision should not be affected by the replication attempt.”
“…for a number of reasons, we decided on an “exact” replication that would make no modifications beyond correcting typos and similar errors–including to the sample size.
“We suggest three funding models going forward. First, experimenters could include in their grants a request for funds to cover a replication attempt by a journal. Second, “open” journals often charge publication fees once an article is accepted for publication. …it is not out of the question to make replication fees a standard part journal publication fees. Third, journals, or the societies that run them, could apply for grants themselves to run journal-based replication pilot programs…”
“…we anticipated the replication would be a success. In that case, we believed that the original paper would be modified to show the estimates of the main treatment effect from each of the original and replication study, and then pool the data for all subsequent analyses.”
“We anticipated there was some chance the replication would be a failure, in which case we presumed that we would make some note of it in the published paper, and then put together a short note with the rest of the results.”
“However, we neglected a third scenario which is, historically, incredibly unlikely: we might get the opposite effect from that found in the paper. This turned out to be what happened.”
“The next section describes in more detail all the things that went wrong (and right) in our attempt at journal-based replication…”
“This paper proposes, and shows a proof-of-concept of, a novel mechanism of ensuring replication: journal-based replication. By making publication decisions before a replication is conducted, it reduces the possibility of “file-drawer” problems.”
“Comparing the model of journal-based replication to a model in which replication is achieved through other means (but is still done), the difference would be that the costs of replication would be borne by the journal, and, more likely, the authors of the original study.”
“Finally, as people wonder what the purpose of journals is in an age of open access…it seems that enforcing replication could be one such purpose.”
To read the article, click here.
[Excerpts taken from the article “Predict science to improve science” by Stefano DellaVigna, Devin Pope, and Eva Vivalt, published in Science]
“Many fields of research, such as economics, psychology, political science, and medicine, have seen growing interest in new research designs to improve the rigor and credibility of research…”
“…relatively little attention has been paid to another practice that could help to achieve this goal: relating research findings to the views of the scientific community, policy-makers, and the general public.”
“We stress three main motivations for a more systematic collection of predictions of research results.”
“…we do not have a systematic procedure to capture the scientific views prior to a study, nor the updating that takes place afterward.”
“…people routinely evaluate the novelty of scientific results with respect to what is known. However, they typically do so ex post, once the results of the new study are known. Unfortunately, once the results are known, hindsight bias (“I knew that already!”) makes it difficult for researchers to truthfully reveal what they thought the results would be. This stresses the importance of collecting systematic predictions of results ex ante.”
“A second benefit of collecting predictions is that they can…potentially help to mitigate publication bias”
“…null results…are rarely published even when authors have used rigorous methods to answer important questions…if priors are collected before carrying out a study, the results can be compared to the average expert prediction, rather than to the null hypothesis of no effect.”
“This would allow researchers to confirm that some results were unexpected, potentially making them more interesting and informative, because they indicate rejection of a prior held by the research community; this could contribute to alleviating publication bias against null results.”
“A third benefit of collecting predictions systematically is that it …may help with experimental design.”
“For example, envision a behavioral research team consulted to help a city recruit a more diverse police department. The team has a dozen ideas for reaching out to minority applicants, but the sample size allows for only three treatments to be tested…the team can elicit predictions for each potential project and weed out those interventions judged to have a low chance of success or focus on those interventions with a higher value…”
“These three broad uses of predictions highlight two important implications. First, it will be important to collect forecast data systematically to draw general lessons.”
“Second, like preanalysis plans, it is critical to set up the collection of predictions before the results are known, to avoid the impact of hindsight bias.”
“With these features in mind, a centralized platform that collects forecasts of future research results can play an important role. Toward this end, in coordination with the Berkeley Initiative for Transparency in the Social Sciences (BITSS), we have developed an online platform for collecting forecasts of social science research results (https://socialscienceprediction.org/).”
“The platform will make it possible to track multiple forecasts for an individual across a variety of interventions, and thus to study determinants of forecast accuracy, such as characteristics of forecasters or interventions, and to identify superforecasters…”
“There are many open questions about the details of the platform…We expect that continued work and experimentation will provide more clarity regarding such design questions.”
To read the article, click here. (NOTE: This article is behind a paywall.)
[Excerpts taken from the article “A Famous Study Found That Blind Auditions Reduced Sexism in the Orchestra. Or Did It?” by Robby Soave, published in Reason Magazine]
“One of the best-known scientific studies to posit that implicit bias—the idea that all people are unconsciously racist, sexist, etc.—can be counteracted via strategic effort is taking a…beating. It now appears that the findings were significantly overstated.”
“The study, “Orchestrating Impartiality: The Impact of ‘Blind’ Auditions on Female Musicians,” by Harvard University’s Claudia Goldin and Princeton University’s Cecilia Rouse, was released in 2000. Its bombshell finding was that blind orchestra auditions—which prevented the choosers from seeing whether each auditioner was male or female—increased female auditioners’ odds by 50 percent.”
“The American Enterprise Institute’s Christina Hoff Sommers notes that the study was ‘lionized by Malcolm Gladwell, extolled by Harvard thought leaders, and even cited in a dissent by Justice Ruth Bader Ginsburg.'”
“In May, Columbia University statistician Andrew Gelman took a deep dive into the study. He described them as “not very impressive at all,” and had great difficulty trying to locate the 50 percent statistic within the modest findings.”
“‘You shouldn’t be running around making a big deal about point estimates when the standard errors are so large,’ he wrote. ‘I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors. But from a modern perspective we can see the problem.’”
“Sommers…noted the existence of another study that had contradicted Goldin and Rouse:”
“In 2017 a team of behavioral economists in the Australian government published the results of a large, randomized controlled study entitled “Going Blind to See More Clearly.” It was directly inspired by the blind-audition study.”
“For the study, more than 2,000 managers in the Australian Public Service were asked to select recruits from randomly assigned résumés—some disguising the applicant’s sex, others not. The research team fully expected to find far more female candidates shortlisted when sex was disguised. But, as the stunned team leader told the local media: ‘We found the opposite, that de-identifying candidates reduced the likelihood of women being selected for the shortlist.'”
“It turned out that many senior managers, aware that sexist assumptions had once kept women out of upper-level positions, already practiced a mild form of affirmative action. Anonymized hiring was not only time-consuming and costly, it proved to be an obstacle to women’s equality.”
“Blind interviews and auditions may be preferable for other reasons. They may even reduce implicit bias in some situations. But as is so often the case, the sweeping claims of social scientists do not seem to survive scrutiny.”
To read the article, click here.
[Excerpts are taken from the article “Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis” by Starns et al., published in Advances in Methods and Practices in Psychological Science]
“The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data.”
“The most direct way to assess inferential skills is to create data sets for which the correct inferences are known and to determine whether researchers can discover these correct inferences through blinded data analysis.”
“In what follows, we demonstrate the blinded inference paradigm with an example study using a recognition memory paradigm.”
“…the first three authors sent recognition memory researchers (the contributors) seven data sets generated with common experimental manipulations and asked them to make inferences about memory performance.”
“In a standard recognition memory task, participants are asked to indicate whether they previously encountered a given stimulus (often a word) in a certain context (typically, a study list). A common question is whether, and to what extent, an independent variable affects discriminability (the ability to distinguish stimuli that were and were not seen in the target context), and in many cases this determination is obscured by differences in response bias (the overall predilection for responding “studied”).”
“Distinguishing memory-discriminability effects from bias effects is a common empirical issue for recognition memory researchers…The available tools to interpret discriminability are well established, and some have been in use for nearly 70 years…”
“Despite these truths, the experts who served as contributors in this study had mixed success…we found surprisingly low accuracy for this relatively simple inferential task…”
“…all the contributors to this study drew inferences about the same data. Thus, the disparate conclusions that they reached are not another example of the replication crisis.”
“…our findings suggest that current efforts to improve research quality are incomplete, in that they largely focus on limiting researchers’ ability to bias results…Even unbiased analysis techniques can be ineffective…”
“Our results suggest that even well-established areas of research may be facing an inference crisis that is every bit as troubling as the replication crisis.”
To read the article, click here. (NOTE: Article is behind a paywall.)
[* EiR = Econometrics in Replications, a feature of TRN that highlights useful econometrics procedures for re-analysing existing research. The material for this blog is primarily drawn from the recent working paper “Difference-in-differences with variation in treatment timing” by Andrew Goodman-Bacon, available from his webpage at Vanderbilt University. FIGURE 1 is modified from a lecture slide by Pamela Jakiela and Owen Ozier.]
In a recent blog at TRN, I discussed research by Clément de Chaisemartin and Xavier D’Haultfoeuille (C&H) that pointed out how heterogeneity in treatment effects causes two-way fixed effects (2WFE) estimation to produce biased estimates of Average Treatment Effects on the Treated (ATT).
This paper by Andrew Goodman-Bacon (GB) provides a nice complement to C&H. In particular, it decomposes the 2WFE estimate into mutually exclusive components. One of these can be used to identify the change in treatment effects over time. An accompanying Stata module (“bacondecomp”) allows researchers to apply GB’s procedure.
In this blog, I summarize GB’s decomposition result and reproduce his example demonstrating how his Stata command can be applied.
Conventional difference-in-differences with homogeneous treatment effects
The canonical DD example consists of two groups, “Treatment” and “Control”, and two time periods, “Pre” and “Post”. The treatment is simultaneously applied to all members of the treatment group. The control group never receives treatment. The treatment effect is homogenous both across the treated individuals and “within” individuals over time. If there are time trends, we assume they are identical across both groups (“common trends assumption”).
FIGURE 1 motivates the corresponding DD estimator.
Let δ be the ATT (which is the same for everybody and constant over time). Note that ATT is given by the double difference DD, where,
The first difference sweeps out any unobserved fixed effects that characterize Treatment individuals. This leaves δ plus the time trend for the Treatment group.
The second difference (in parentheses) sweeps out unobserved effects associated with Control individuals. This leaves the time trend for the Control group.
The first difference minus the second difference then leaves δ, the ATT, assuming both groups have a common time trend. (Note how the “common trends” assumption is key to identifying δ.)
It is easily shown that, given the above assumptions, that OLS estimation of the regression specification below produces an unbiased estimate of δ.
A more realistic, three-period example
Now consider a more realistic example, close in spirit to what researchers actually encounter in practice. Let there be three groups, “Early Treatment”, “Late Treatment” and “Never Treated”; and three time periods, “Pre”, “Mid”, and “Post”.
FIGURE 2 motivates the following discussion.
The Early Treatment group receives treatment at t*k (GB uses the k subscript to indicate early treatees).
The Late Treatment group receives treatment at t*l , t*l > t*k .
Suppose a researcher were to estimate the following 2WFE regression equation, where Dit is a dummy variable indicating whether individual i was treated at or before time t. For example, Dit = 0 and 1 for Late treatees at times “Mid” and “Post”, while Dit = 1 for Early treatees at times “Mid” and “Post”,
GB shows that the OLS estimate of βDD is a weighted average of all possible DD paired differences. One of those paired differences (cf. Equation 6 in GB) is
Note that in this case, the Early Treatment group (subscripted by k) can serve as a control group for the Late Treatment group because its treatment status does not change over the “Mid”/“Post” period. This particular paired difference ends up being important.
GB goes on to derive the following decomposition result: The probability limit of the OLS estimator of βDD consists of three components:
VWATT is the Variance Weighted ATT, VWCT is the Variance Weighted Common Trends, and ΔATT is the change in individuals’ treatment effects that occurs over time, where the weights come from sample size and treatment variance.
When the common trends assumption is valid (VWCT=0), and the treatment effect is homogeneous both across individuals and within individuals over time, then the probability limit equals δ, the homogeneous treatment effect.
However, if treatment effects are heterogeneous, then even if the common trends assumption holds, estimation of the 2WFE specification will not equal the ATT. There are two sources of bias.
The first bias arise because OLS weights individual treatment effects differently depending on (i) the number of people who are treated and (ii) the timing of the treatment. This will introduce a bias if the size of the treatment effect is associated with either of these. However, this bias is not necessarily a bad thing. It is the byproduct of minimizing the variance of the estimator, so there are some efficiency gains that accompany this bias.
The second bias is associated with changes in the treatment effect over time, ΔATT. It’s entirely a bad thing.
Consider again the paired difference
The second term is the difference in outcomes for the Early treatees between the Post and Mid periods. Because Early treatees are treated for both of these periods, this difference should sweep away everything but the time trend if the treatment effect stays constant over time.
However, if treatment effects vary over time, say the benefits depreciate (or, alternatively, accumulate), the treatment effect will not be swept out. As a result, the change in the treatment effect will carry through to the respective DD estimate. As a result, the DD estimate will respectively over- or under-estimate the true treatment effect.
GB’s decomposition allows one to investigate this last type of bias. Towards that end, GB (along with Thomas Goldring and Austin Nichols) has written a Stata module called bacondecomp.
Application: Replication of Stevenson and Wolfers (2006)
To demonstate bacondecomp, GB replicates a result from the paper “Bargaining in the Shadow of the Law: Divorce Laws and Family Distress” by Betsey Stevenson and Justin Wolfers (S&W), published in The Quarterly Journal of Economics in 2006.
Among other things, S&W estimate the effect of state-level, no-fault divorce laws on female suicide. Over their sample period of 1964–1996, 37 US states gradually adopted no-fault divorce. 8 states had already done so, and 5 states never did.
[NOTE: The data and code to reproduce the results below are taken directly from the examples in the help file accompanying bacondecomp. They can be obtained by installing bacondecomp and then accessing the help documentation through Stata].
GB does not exactly reproduce S&W’s result, but uses a similar specification and obtains a similar result. In particular, he estimates the following regression
where asmrs is the suicide mortality rate per one million women; post is the treatment dummy variable (i.e., Dit); pcinc, asmrh, and cases are control variables for income, the homicide mortality rate, and the welfare case load; and αi and αt are individual and time fixed effects.
The 2WFE estimate of βDD is -2.52. In other words, this specification estimates that no-default divorce reform reduced female suicides by 2.52 fatalities per million women.
The bacondecomp command decomposes the 2WFE estimate of -2.52 into three separate components by treatment (T) and control (C) groups.
Timing_groups: Early treatees (T) versus Late treatees (C) & Late treatees (T) versus Early treatees (C).
Always_v_timing: Treatees (T) versus Always treated/Pre-reform states (C).
And Never_v_timing: Treatees (T) versus Never treated (C).
bacondecomp produces the following table, where “Beta” is the DD estimate for the respective group and “TotalWeight” represents its share in the overall estimated effect (-2.52). Notice that the sum of the products of “Beta” × “TotalWeight” ≈ the 2WFE estimate.
Conspiculously, the first group (Timing_groups) finds that no-fault divorce reform is associated with an increase in the female suicide rate (+2.60). In contrast, the latter two groups find a decrease (-7.02 and -5.26). This is indicative that there may be changes in treatment effects over time. If so, this would invalidate the difference-in-differences estimation framework.
Unfortunately, bacondecomp does not produce a corrected estimate of ATT. It is primarily useful for identifying a potential problem with time-varying treatment effects. As a result, it should be seen as complementing other approaches, such as the estimation procedures of de Chaisemartin and D’Haultfoeuille (see here), or an alternative approach such as an event study framework that includes dummies for each post-treatment period (see here).
Bob Reed is a professor of economics at the University of Canterbury in New Zealand. He is also co-organizer of the blogsite The Replication Network. He can be contacted at bob.reed@canterbury.ac.nz.
References
Goodman-Bacon, A., Goldring, T., & Nichols, A. (2019). bacondecomp: Stata module for decomposing difference-in-differences estimation with variation in treatment timing. https://ideas.repec.org/c/boc/bocode/s458676.html
[Excerpts taken from the report “Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary Report” by Daniel Lakens et al., posted at MetaArXiv Preprints ]
“…given the broad array of problems that make it difficult to evaluate the evidential value of single studies, it seems more imperative than ever to use meta-analytic techniques…”
“…some researchers have doubted whether meta-analyses can actually produce objective knowledge when the selection, inclusion, coding, and interpretation of studies leaves as least as much room for flexibility in the analysis and conclusions …as is present in single studies…”
“Our goals were to…assess the similarity between published and reproduced meta-analyses and identifying specific challenges that thwart reproducibility…”
“We collected all (54) meta-analyses that were published in Psychological Bulletin, Perspectives on Psychological Science, and Psychonomic Bulletin & Review in 2013 and 2014.”
“From these 54 meta-analyses, only…67% or the meta-analyses (36) included a table where each individual effect size, and the study it was calculated from, was listed.”
“Twenty out of the 54 meta-analyses were randomly selected to be reproduced. Teams attempted to reproduce these meta-analyses as completely as possible based on the specifications in the original articles.”
“Five meta-analyses proved so difficult to reproduce that is was not deemed worthwhile to continue with the attempt.”
“For these five meta-analyses, two did not contain a table with all effect sizes, making it impossible to know which effects were coded, and to check whether effects were coded correctly. Two other meta-analyses could not be reproduced from articles in the published literature because the relevant information was missing, and thus needed access to raw data, and in another meta-analysis not enough details were provided on how effect sizes were selected and calculated.”
“Coding for 9 meta-analyses was completed, and coding for 6 metaanalyses is completed to some extent, but still ongoing.”
“All reproduced meta-analyses can be found on the Open Science Framework: https://osf.io/q23ye/files/.”
“Overall, there was unanimous agreement across all seven teams involved in extracting data from the literature that reproducing published meta-analyses was much more difficult than we expected.”
“Many effect sizes in meta-analyses proved impossible to reproduce due to a combination of a lack of available raw data, difficulty locating the relevant effect size from the article, unclear rules about how multiple effect sizes were combined or averaged, and lack of specification of the effect size formula used to convert effect sizes.”
“A frequent source of disagreement between the meta-analysis and the recoded effect sizes concerned the sample size in the study, or the sample sizes in each group…sample sizes are used to calculate the variance for effect sizes…due to the fact that sample sizes for specific analyses are often not clearly provided in original articles, this information turned out to be surprisingly difficult to code.”
“Doing a meta-analysis is a great way to learn which information one needs to report to make sure the article can be included in a future meta-analysis.”
“Similarly, reproducing a meta-analysis is a great way to learn which information one needs to report in a meta-analysis to make sure a meta-analysis can be reproduced.”
“We can highly recommend researchers interested in performing a meta-analysis to start out by (in part) reproducing a published meta-analysis.”
“Because meta-analyses play an important role in cumulative science, they should be performed with great transparency, and be reproducible. With this project we hope to have provided some important preliminary observations that stress the need for improvement, and provided some practical suggestions to make meta-analyses more reproducible in the future.”
To read the report, click here.
[Excerpt taken from the article “What’s next for Registered Reports?” by Chris Chambers, published in Nature]
“For the past six years, I have championed Registered Reports (RRs), a type of research article that is radically different from conventional papers. The 30 or so journals that were early adopters have together published some 200 RRs, and more than 200 journals are now accepting submissions in this format (see ‘Rapid rise’).”
“With RRs on the rise, now is a good time to take stock of their potential and limitations.”
HOW DO THEY WORK?
“The Registered Report format splits conventional peer review in half. First, authors write an explanation of how they will probe an important question. …Before researchers do the studies, peer reviewers assess the value and validity of the research question, the rationale of the hypotheses and the rigour of the proposed methods. They might reject the Stage 1 manuscript, accept it or accept it pending revisions to the study design and rationale. This ‘in-principle acceptance’ means that the research will be published whatever the outcome, as long as the authors adhere closely to their protocol and interpret the results according to the evidence.”
“After the Stage 1 manuscript is accepted, the authors formally preregister it in a recognized repository such as the Open Science Framework, either publicly or under a temporary embargo. They then collect and analyse data and submit a completed ‘Stage 2’ manuscript that includes results and a discussion.”
“The Stage 2 submission is sent back to the original reviewers, who cannot question the study rationale or design now that the results are known. Whether the results are judged by reviewers to be new, groundbreaking or exciting is irrelevant to acceptance.”
“An analysis this year suggests that RRs are more likely to report null findings than are conventional articles: 66% of RRs for replication studies did not support initial hypotheses; for RRs of novel studies, the figure was 55%. Estimates for conventional papers range from 5 to 20%.”
MOVING FORWARD
“RRs are not a panacea — the format needs constant refinement. It currently sits rather awkwardly between the old world of scientific publishing and the new. Innovations over the next few years should make this format even more powerful, and stimulate wider reforms.”
“Transparency. When RRs first launched, some journals published Stage 2 manuscripts but not those for Stage 1, making it impossible for readers to see whether the completed protocol matched the planned one. In 2018, the Center for Open Science launched a simple tool that places submitted Stage 1 manuscripts in a public registry. This is now used by many journals…”
“Standardization. …Currently, submitted manuscripts are often prepared in word-processing software and contain insufficient methodological detail or linking between predictions and analyses. The next generation of RRs — ‘Registered Reports 2.0’ — is likely to be template-based and could integrate tools such as Code Ocean. This would ensure that analyses are immutable within a stable, self-contained software environment. With standardized metadata and badging, RRs will become useful for systematic reviews and meta-analyses.”
“Efficiency. The review process can be extended even further back in the research life cycle. Under the emerging RR grant model, reviewers award funding and signal in-principle acceptance of a research publication simultaneously or in rapid succession. The Children’s Tumor Foundation and PLoS ONE have pioneered such a partnership, as have Cancer Research UK and the journal Nicotine & Tobacco Research. More are in the works.”
“The lesson of RRs speaks to all areas of science reform.…Instead of pitting what is best for the individual against what is best for all, create a model that benefits everyone — the scientist, their community and the taxpayer — and the rest will come naturally.”
To read the article, click here.
[Excerpts taken from the article “Research on research gains steam” by Dalmeet Singh Chawla, published in Chemical and Engineering News]
“The use of scientific methodology to study science itself is called metascience. The discipline has become mainstream in recent years, tackling some of the thorniest problems science faces, including a lack of reproducibility of academic literature, biases in peer review, and the fair allocation of research funding.”
“On Sept. 30, metaresearch got another boost when an international coalition of policymakers, funders, universities, publishers, and researchers launched the Research on Research Institute (RoRI), which will be dedicated to tackling metascience questions on a mass scale.”
“James Wilsdon, a research policy scholar at the University of Sheffield and the institute’s founding director, says that the scientific community is “woefully underinvesting” in metascience given the benefits it could have for the “efficiency, dynamism and sustainability” of the research enterprise.”
“The four founding partners of the institute are the biomedical research charity Wellcome Trust, Leiden University’s Centre for Science and Technology Studies, research technology firm Digital Science, and the University of Sheffield.”
“Wilsdon says a consortium of additional strategic partners, including public and private organizations from 11 nations, are already onboard, and conversations are underway with more potential partners.”
“The three broad headings under which RoRI is spearheading its initial projects are decision-making, careers, and culture. Wilsdon thinks the first few initiatives are likely to focus on investigating how funding applications pass peer review and how research money is allocated, including testing different funding models.”
To read the article, click here.
[Excerpts taken from the article “Economists run experiments, too” by Aiora Zabala, published at Nature Sustainability]
“For most who don’t know this literature, it’s easy to lightly recommend that we should implement taxes, subsidies or regulations to tackle environmental challenges…Economists and behavioural scientists are trying to find the answer by running experiments (much like in life sciences labs). With these, researchers test policy variations before they are implemented…”
“Two years ago, scholars concerned with these questions formed the Research Network on Economic Experiments for the Common Agricultural Policy (REECAP). In early September this year, they gathered in Osnabrück (Germany) to discuss their latest studies, future directions for the field and common concerns.”
“Among the sessions, they included speakers at the interface between science and policy, directly advising European institutions on their agricultural policies, doing environmental lobbying in Brussels, or directly making decisions, for example.”
“Another topic concerned the so-called crisis of reproducibility of published studies, first discussed in psychology and in life sciences, and put on the spot among economists by studies like this and this.”
“To a panel on the topic, I brought the view on the challenges and opportunities of such crisis from an editors’ perspective. Some areas for action are policy checklists and code and data statements (as currently done across Nature journals).”
To read the article, click here.
You must be logged in to post a comment.