[Excerpts taken from the article, “Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)”, by Hardwicke, et al., posted at MetaArXiv Preprints]

“…we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related indicators in psychology articles published between 2014-2017.”

“The indicators were open access to published articles; availability of study materials, study protocols, raw data, and analysis scripts; preregistration; disclosure of funding sources and conflicts of interest; conduct of replication studies; and cumulative synthesis of evidence in meta-analyses and systematic reviews.”

“…we used a random number generator to sample 250 articles from all 224,556 documents in the Scopus database (as of 22nd September, 2018) designated with the document type “article” or “review”, published between 2014 and 2017, and with an All Science Journal Classification (ASJC) code related to psychology…”

Article availability (‘open access’)

“Among the 237 English-language articles, we were able to obtain a publicly available version for 154…whereas 74…were only accessible to us through a paywall and 9…were not available to us at all…”

Materials and protocol availability

“Of the 183 articles involving primary data…26 contained a statement regarding availability of original research materials such as survey instruments, software, or stimuli…”

“Of the 188 articles involving primary or secondary data, zero reported availability of a study protocol…”

“For the 26 articles where materials were reportedly available, the materials were not actually available for 7 articles due to broken links.”

Data availability

“Of the 188 articles that involved primary or secondary data, 4 contained data availability statements…”

“For one dataset, a fee was required (which we did not pay) to obtain access. Of the 3 accessible datasets…One dataset was incomplete whereas the remaining 2 appeared complete and clearly documented.”

Analysis script availability

“Of the 188 articles that involved primary or secondary data, an analysis script was shared for one article…”

Pre-registration

“Of the 188 articles involving primary or secondary data, 5 included a statement regarding preregistration…All accessible pre-registrations contained information about hypotheses and methods but did not contain analysis plans.”

Replication and evidence synthesis

“Of the 188 articles involving primary or secondary data, 10…claimed to include a replication study. Of the 183 articles involving primary data, 1 article…was cited by another article that claimed to be a replication.”

“Of the 183 articles involving primary data, 21 were formally included in systematic reviews…12 were formally included in meta-analyses…”

Discussion

“Our evaluation of transparency and reproducibility-related research practices in a random sample of 250 psychology articles published between 2014-2017 highlights that whilst many articles are publicly available, crucial components of research – protocols, materials, raw data, and analysis scripts – are rarely made publicly available…”

“Pre-registration remains a nascent proposition with minimal adoption.”

“Replication or evidence synthesis via meta-analysis or systematic review is relatively infrequent.”

“The current study has several caveats and limitations. Firstly, our findings are based on a random sample of 250 articles, and the obtained estimates may not necessarily generalize to specific contexts, such as other disciplines…”

To read the article, click here.