[Excerpts taken from the article “No Data in the Void: Values and Distributional Conflicts in Empirical Policy Research and Artificial Intelligence” by Maximilian Kasy, published at econfip.org]

“Decision making based on data…is becoming ever more widespread. Any time such decisions are made, we need to carefully think about the goals we want to achieve, and the policies we might possibly use to achieve them.”

“…Let us now…turn to the debates about publication bias, replicability, and the various reform efforts aimed at making empirical research in the social and life sciences more credible.”

“…what it is that reforms of academic research institutions and norms wish to ultimately achieve: What is the objective function of scientific research and publishing?”

“Consider, as an example, clinical research on new therapies. Suppose that in some hypothetical area of medicine a lot of new therapies, say drugs or surgical methods, are tested in clinical studies.”

“In this hypothetical scenario, which findings should be published? That is, which subset of studies should doctors read? In order to improve medical practice, it would arguably be best to tell doctors about the small subset of new therapies which were successful in clinical trials.”

“If this is the selection rule used for publication, however, published findings are biased upward. Replications of the published clinical trials will systematically find smaller positive effects or even sometimes negative effects.”

“This reasoning suggests that there is a deep tension between relevance (for decision making) and replicability in the design of optimal publication rules.”

“In Frankel and Kasy (2018), Which findings should be published?, we argue that this type of logic holds more generally, in any setting where published research informs decision makers and there is some cost which prevents us from communicating all the data. In any such setting, it is optimal to selectively publish surprising findings.”

“These considerations leave us with the practical question of what to do about the publication system…A possible solution might be based on a functional differentiation of publication outlets”

“There might be a set of top outlets focused on publishing surprising (“relevant”) findings…These outlets would have the role of communicating relevant findings to attention-constrained readers (researchers and decision makers). A key feature of these outlets would be that their results are biased, by virtue of being selected based on surprisingness.”

“There might then be another, wider set of outlets that are not supposed to select on findings.., For experimental studies, pre-analysis plans and registered reports (results-blind review) might serve as institutional safe-guards to ensure the absence of selectivity by both researchers and journals.”

“Journals that explicitly invite submission of “null results” might be an important part of this tier of outlets. This wider set of outlets would serve as a repository of available vetted research, and would not be subject to the biases induced by the selectivity of top-outlets.”

To read the article, click here.