[Excerpts are taken from the article “Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis” by Starns et al., published in Advances in Methods and Practices in Psychological Science]

“The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data.”

“The most direct way to assess inferential skills is to create data sets for which the correct inferences are known and to determine whether researchers can discover these correct inferences through blinded data analysis.”

“In what follows, we demonstrate the blinded inference paradigm with an example study using a recognition memory paradigm.”

“…the first three authors sent recognition memory researchers (the contributors) seven data sets generated with common experimental manipulations and asked them to make inferences about memory performance.”

“In a standard recognition memory task, participants are asked to indicate whether they previously encountered a given stimulus (often a word) in a certain context (typically, a study list). A common question is whether, and to what extent, an independent variable affects discriminability (the ability to distinguish stimuli that were and were not seen in the target context), and in many cases this determination is obscured by differences in response bias (the overall predilection for responding “studied”).”

“Distinguishing memory-discriminability effects from bias effects is a common empirical issue for recognition memory researchers…The available tools to interpret discriminability are well established, and some have been in use for nearly 70 years…”

“Despite these truths, the experts who served as contributors in this study had mixed success…we found surprisingly low accuracy for this relatively simple inferential task…”

“…all the contributors to this study drew inferences about the same data. Thus, the disparate conclusions that they reached are not another example of the replication crisis.”

“…our findings suggest that current efforts to improve research quality are incomplete, in that they largely focus on limiting researchers’ ability to bias results…Even unbiased analysis techniques can be ineffective…”

“Our results suggest that even well-established areas of research may be facing an inference crisis that is every bit as troubling as the replication crisis.”

To read the article, click here. (NOTE: Article is behind a paywall.)