[From the paper “Preregistration and reproducibility” by Eirik Strømland, forthcoming in the Journal of Economic Psychology]

“I present a simple model in which the research community is populated by agents who may report their results unconditionally or conditionally on statistical significance and show that greater reliance on preregistration improves the estimation of the effect sizes through increasing the share of “frequentist” researchers. As replicators are likely to estimate statistical power based on the published effect sizes, preregistration is therefore also expected to improve reproducibility rates.”

“Table 1 shows that in a low-powered setting, the inflation bias is generally sizeable unless the share of frequentist researchers is very high. In the worst-case scenario, the reported effect will be, on average, 2.2 times as high as the true effect, and researchers who estimate the power of a replication study based on this effect will set a sample size that in reality gives them only 24% power.”

“The results shown in Table 1 depend on the assumptions made about statistical power. If the statistical power in the original study is high (80%), then the inflation bias will generally be very small even in the worst-case scenario with only selective reporting. The reason is that for a constant and conventional test statistics (e.g. the t-test), an increase in sample size leads to higher values of the test statistic for lower values of the estimate. The truncated sampling distribution would then approach the “frequentist” sampling distribution … and even replicators who base their power estimates on selectively reported estimates would obtain quite accurate power estimates.”

“However, … as long as the statistical power is below about 50%, the sampling distribution of only statistically significant effects will be different enough from the unconditional sampling distribution that the difference between the two distributions will be large in practice.”

To read the article, click here (NOTE: The article is behind a paywall.)