A Random Sampling of the State of Transparency and Reproducibility in Social Science Journals

[From the preprint “An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017)” by Tom Hardwicke, Joshua Wallach, Mallory Kidwell, & John Ioannidis posted at MetaArXiv Preprints]
“In this study, we evaluated a broad range of indicators related to transparency and reproducibility in a random sample of 198 articles published in the social sciences between 2014 and 2017.”
“…sample characteristics are displayed in Table 2.”TRN1(20190507)
“Among the 198 eligible articles, 95…had a publicly available version (Fig. 1A) whereas 84…were only accessible to us through a paywall, …19…additional articles were not available to our team, highlighting that even researchers with broad academic access privileges cannot reach portion of the scientific literature.”TRN2(20190507)
“Raw data are the core evidence that underlies scientific claims. However, the vast majority of 103 articles we examined did not contain a data availability statement (Fig. 1D). Eight articles stated that they had used an external data source but it was unclear if the data were available. Among a further 8…datasets that were reportedly available, 2 were reportedly ‘available upon request’ from the authors and 2 had broken links to supplementary materials and a personal/institutional webpage. Of the 4 accessible datasets…2 were both incomplete and not clearly documented.” TRN3(20190507)
“Analysis scripts provide detailed step-by-step descriptions of performed analyses, often in the form of computer code (e.g., R, Python, or Matlab) or syntax (SPSS, Stata, SAS). Although 3 of 103 … articles reported that analysis scripts were available (Fig. 1E), 1 of these involved a broken link to journal-hosted supplementary information and 1 was only “available on request”, which we did not attempt to confirm. The 1 available analysis script was provided in journal-hosted supplementary information.”TRN4(20190507)
“Replication studies repeat the methods of previous studies in order to systematically gather evidence on a given research question. Evidence gathered across multiple relevant studies can be formally collated and synthesized through systematic reviews and meta-analyses. Only 2 of the 103 … articles we examined explicitly self-identified as a replication study.”
“Pre-registration refers to the archiving of a read-only, time-stamped study protocol in a public repository (such as the Open Science Framework) prior to study commencement….None of the articles specified that the study was pre-registered…”
“Our empirical assessment of a random sample of articles published between 2014 and 2017 suggests a serious neglect of transparency and reproducibility in the social sciences. Most research resources, such as materials, protocols, raw data, and analysis scripts, were not explicitly available, no studies were pre-registered, disclosure of funding sources and conflict of interests was modest, and replication or evidence synthesis via meta-analysis or systematic review was rare.”
To read the article, click here.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: