[Excerpts taken from the article “In Praise of Confidence Intervals” by David Romer, posted at the American Economic Association’s 2020 annual conference website]

“…most modern empirical papers concentrate on two characteristics of their findings: whether the point estimates are statistically significantly different from zero, and the economic interpretation of the point estimates.”

“But in almost all applications, there are potential values of the parameters other than the point estimates and zero that are of interest. Focusing on point estimates and statistical significance obscures the implications of the findings for those values.”

“When a paper only reports that an estimate is significantly different from zero…very little is conveyed about whether the t-statistic is…moderately above 2…or far above 2…smaller t-statistics provide only moderate evidence against values of the parameter that are economically small.”

“Although the key point is that reporting and discussing confidence intervals would often convey much more about the implications of papers’ findings than the usual approach, it is not obvious that the usual 2-standard error confidence interval is the best choice.”

“A common shortcut way of interpreting a confidence interval is that the results provide strong evidence against parameter values outside the interval and are essentially equally supportive of all values inside it.”

“…suppose initially researchers viewed the point estimate and a value at the boundary of the 2-standard error confidence interval as equally likely. Ex post (assuming normality for simplicity), they should view the point estimate as roughly 7 times as likely as the value at the boundary. That is, even though both values are in the confidence interval, the results are considerably more supportive of the point estimate than of the value at the boundary.”

“Even better would be to report both 1-standard error and 2-standard error bands for papers’ key estimates (something that is now sometimes done in figures).”

“Ex post, a researcher who started with flat priors would view the point estimate as only moderately more likely than the other values in the 1-standard error band (…considerably less than twice as likely).”

“…in this case, the natural, and roughly correct, shortcut interpretation would be that the results provide little information about the relative merits of different values within the 1-standard error interval, moderate evidence against values in the 2-standard error but not the 1-standard error interval relative to the point estimate, and strong evidence against values outside the 2-standard error band relative to the point estimate.”

“Consider two possible papers estimating the multiplier for government purchases. In both, the point estimate is 3.0. In one, the standard error is 1.3, while in the other it is 0.7.”

“With the usual current approach to discussing empirical results, the two papers would describe their findings in similar terms. Both would observe that the estimate is statistically significant and would focus on the economic interpretation of a multiplier of 3.”

“In fact, however, the two results would have very different implications for most questions about the multiplier economists are interested in.”

“The one that obtained a standard error of 1.3 (implying a 1-standard error confidence interval of (1.7, 4.3) and a 2-standard error interval of (0.4, 5.6)) would, as before, observe that the hypothesis of a multiplier of zero is rejected. But it would go on to emphasize that the estimate was not very precise: the results provide little evidence against more conventional values of the multiplier such as 1.8, only moderate evidence against a multiplier of 1 or slightly below, and very strong but not overwhelming evidence against a multiplier of zero.”

“The paper with a standard error of 0.7 (and thus a 1-standard error confidence interval of (2.3, 3.7) and a 2-standard error interval of (1.6, 4.4)), in contrast, would observe that a multiplier of zero was not merely rejected, but rejected overwhelmingly, And it would proceed to point out that the results provide quite strong evidence against not just zero but against values of 1 and below, and that they even provide moderately strong evidence against recent estimates in the vicinity of 1.8.”

“All this could be accomplished with the addition of a sentence or two reporting confidence intervals, a few adjectives (“imprecise,” “overwhelming,” “moderate,” and so on), and a few sentences describing the confidence intervals’ implications for key candidate values of the multiplier.”

“…reporting the confidence intervals for papers’ focal estimates and briefly discussing their key implications would add only trivially to papers’ length, and would often greatly increase the amount of information conveyed about the economic implications of the findings.”

To read the article, click here.