[Excerpts taken from the working paper “Practical Significance, Meta-Analysis and the Credibility of Economics” by Tom Stanley and Chris Doucouliagos, posted at SSRN]

“…we find that large biases and high rates of false positives will often be found by conventional meta-analysis methods. Nonetheless, the routine application of meta-regression analysis and considerations of practical significance largely restore research credibility.”

“In this study, we employ Monte Carlo simulations to investigate whether typical levels of statistical power, selective reporting, and heterogeneity found in economics research will cause meta-analysis to have notable biases and high rates of false positives; that is, claiming the presence of economic effects or phenomena that may not exist.”

“Our simulations evaluate the performance of four methods: random-effects (RE), unrestricted weighted least squares (WLS), the weighted average of the adequately powered (WAAP), and the PET-PEESE.”

“Recently, Ioannidis et al. (2017) conducted a large survey of bias and statistical power among more than 64,000 reported economic effects from nearly 6,700 research papers. The average number of estimated effects reported per meta-analysis is just over 400…the typical relative heterogeneity (I2) is 93%, and the median exaggeration of reported effects is 100%…we focus on a 50% incidence of selective reporting…”

“To calibrate our simulations, we focus on the 35 meta-analyses of elasticities from Ioannidis et al. (2017) and force the distribution of SEs in the simulations to reproduce closely the distribution of SE found in these 35 reference meta-analyses.”

“…when there is the typical amount of heterogeneity…but no overall effect, the average study reports an elasticity just over 0.18…As the true elasticity gets larger, this bias decreases…but notable bias remains even when the true elasticity is 0.3. These biases are especially large at the highest levels of heterogeneity (I2 = 98%).”

“…All meta-analysis methods fail to distinguish a genuine effect from the artefact of publication bias reliably under common conditions found in economics research. The rate of false positives revealed in our simulations is a serious problem that threatens the scientific credibility and practical utility of simple meta-analysis.”

“To investigate likely departures from the random-effects constant-variance, additive heterogeneity model, we conduct alternative simulation experiments where random heterogeneity is roughly proportional to the random sampling error variance…”

“When heterogeneity is roughly proportional to SE, the simple mean and RE have even larger biases, but the biases of WLS, WAAP and PET-PEESE are much smaller and practically insignificant…”

“…When meta-analysts test for practical significance and heterogeneity is proportional to sampling errors, then false positives are no longer an issue for WLS, WAAP and PET. Unfortunately, random-effects can still have unacceptable rates of false positives even when testing for practical significance.”

“…if meta-analysts use cluster-robust standard errors when they test for practical significance (even with additive, constant-variance heterogeneity), PET has acceptable type I error rates…Note further that WLS, WAAP and PET maintain high levels of power to detect even small elasticities for areas of research which have the typical number of estimates…”

“We take the issue of false positives seriously and, therefore, recommend that systematic reviews and meta-analyses test against practical significance. Doing so largely reduces PET’s type I error rate to acceptable levels for common research conditions in economics.”

To read the paper, click here.