Meta-Research: The Big Picture

[From the paper “Calibrating the scientific ecosystem through meta-research” by Tom Hardwicke, Stylianos Serghiou, Perrine Janiaud, Valentin Danchev, Sophia Crüwell, Steven Goodman, and John Ioannidis, forthcoming in Annual Review of Statistics and Its Application]
“Meta-research has been defined as ‘the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives.'”
“We will map the endeavor of meta-research using the translational framework depicted in Figure 1. This framework is not necessarily comprehensive and aims to be descriptive rather than prescriptive. The goal is to highlight how individual meta-research projects form part of a broader effort to continuously calibrate the scientific ecosystem towards higher standards of efficiency, credibility, and quality.”
TRN(20190704)
Outline
1. IDENTIFYING PROBLEMS
1.1 Incentives and norms
1.2 Lack of transparency
1.3 Statistical schools of thought and statistical misuse
1.4 Reproducibility
2. INVESTIGATING PROBLEMS
2.1 Incentive structures
2.2 Publication bias and selective reporting
2.3 Transparency of research resources
2.4 Suboptimal research design
2.5 Statistical misuse
2.6 Reproducibility
3. DEVELOPING SOLUTIONS
3.1 Journal, funder, society and university policies
3.2 Pre-registration and registered reports
3.3 Reporting guidelines
3.4 Peer review
3.5 Collaboration
3.6 Statistical reform
3.7 Evidence synthesis
4. EVALUATION SOLUTIONS
4.1 Journal policy
4.2 Reporting guidelines
4.3 Pre-registration and registered reports
To read the article, click here.

One Comment on “Meta-Research: The Big Picture

  1. This study cites a previous one twice that was also discussed here https://replicationnetwork.com/2019/05/07/a-random-sampling-of-the-state-of-transparency-and-reproducibility-in-social-science-journals/
    and that as I already pointed out is based on an sample of studies that is alarmingly small and completely ill-suited for the chosen subject of “social sciences” as a very large part of the studies investigated are from journals that do not cover social sciences at all. I recommend a correction.
    Hardwicke TE, Wallach JD, Kidwell MC, Ioannidis, JPA (2019). An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017). MetaArXiv Preprints. Retrieved from: https://doi.org/10.31222/osf.io/6uh

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: