[From the article “The quest for an optimal alpha” by Jeff Miller and Rolf Ulrich, published in PLOS One]

“The purpose of the present article is to show exactly what is necessary to provide a principled justification for a particular α level. …we identify the parameters of a research scenario that must be considered when choosing the optimal α level …We conclude that no definitive case for any particular α level has yet been made, because advocates of particular α levels have never specified—even approximately—the key research parameters whose values are needed to identify the optimal α.”

“Fig 1 illustrates how these rates change with the researcher’s α level, showing results for two different sample sizes (ns), three different effect sizes (d), and a wide range of base rates (π). Critically, for every combination of sample size, effect size, and base rate, the rate of FPs [False Positives] is higher with α = 0.05 than with α = 0.005. In contrast, the rate of FNs [False Negatives] is always higher for α = 0.005 than for α = 0.05. Thus, these two types of decision errors trade off against one another as α changes, and a quantitative model incorporating the frequencies and costs of these errors must be used to choose α.”

“In the end, the question of which α level researchers should use simply cannot be answered without a detailed quantitative model incorporating not only the researcher’s choices of α level and sample size, but also the underlying characteristics of the research scenario and the costs and benefits of reaching the different possible correct and incorrect conclusions. To that end, traditional statistical decision models can be adapted to models of the research process, and we suggest that advocates of any particular α level should use such models—in conjunction with estimates of base rates and payoffs—to give their arguments a firm objective foundation.”

To read the article, click here.