Searching for Specification Searching in Development Impact Evaluations
[From the paper “Specification Searching and Significance Inflation AcrossTime, Methods and Disciplines” by Eva Vivalt, published in the Oxford Bulletin of Economics and Statistics]
“…This paper leverages a database of impact evaluation results collected by AidGrade, a US non-proﬁt organization that focuses on gathering the results of impact evaluations…”
“…This paper examines speciﬁcation searching by comparing the number of barely signiﬁcant results with the number of barely insigniﬁcant results around the conventional cut-off signiﬁcance level of 5%.”
“…This paper ﬁnds that studies using randomized experiments exhibit less speciﬁcation searching than those that do not.”
“…these biases appear less pronounced than has previously been found in some of the other social sciences, and there appears to be little difference between papers written by researchers in economics-related disciplines and papers written by researchers in other ﬁelds like public health.”
“…A second contribution is that speciﬁcation searching is shown to not be static, but a bias that evolves. In particular, RCTs have exhibited signiﬁcantly less bias over time, while quasi-experimental studies have, if anything, exhibited more pronounced biases over time.”
To read the article, click here.