[Excerpts are taken from the article “Affirmative citation bias in scientific myth debunking: A three-in-one case study” by Kåre Letrud and Sigbjørn Hernes, published in PLOS One]

“…we perform case studies of the academic reception of three articles critical of the widely cited yet contentious Hawthorne Effect. By consulting papers citing these critical works, we seek to establish whether, and to what degree the accumulated citations are indeed skewed in favor of the Hawthorne Effect, suggesting the existence of an affirmative citation bias.”

“The idea of a Hawthorne Effect originated from studies on workplace behavior at the Western Electric Company’s Hawthorne Plant during the 1920s and 1930s…Surprisingly, both higher and lower lighting levels supposedly led to increased productivity…‘The Hawthorne Effect’ is now an ambiguous and vague, yet widely used, term, primarily associated with an observer effect: subjects altering their behavior when aware of being observed…”

“We based the case studies on articles arguing against the Hawthorne Effect, interpreted as various observer effects. The selection criteria being that they were unequivocally critical of the effect, that their argumentation was substantial, and that they were extensively cited by peer reviewed articles.”

“Case 1: Franke and Kaul 1978: Franke and Kaul perform the first statistical analysis of the Hawthorne Studies data, and draws conclusions ‘different from those heretofore drawn’…17 articles cited Franke and Kaul, while taking a negative stance towards the Hawthorne Effect, and 63 were neutral…197 affirmed the Hawthorne Effect, and of these 189 cited Franke and Kaul as affirming the Hawthorne Effect.”

“Case 2: Jones 1992: Jones performs an analysis of the data from the relay studies, searching for evidence of the Hawthorne Effect (interpreted as the subjects being aware changes in experimental conditions before or during the experimental period), and finds none. His conclusion: … ‘I must conclude that there is slender or no evidence of a Hawthorne effect…’”

“Consulting these 140 articles we found that 19 were neutral on the matter. 18 cited Jones while criticizing the Hawthorne Effect, whereas 103 affirmed its validity. Of the affirmative articles, 60 cited Jones as affirming the Hawthorne Effect.”

“Case 3: Wickström and Bendix 2000: Citing former reanalyses, Wickström and Bendix…argue that the original Hawthorne Studies did not show adequate evidence of the effect.”

“We were able to retrieve the text of 196 of 198 titles citing Wickström and Bendix published between 2001 and 2018…Merely five articles took a critical stance towards the Hawthorne Effect. 23 were neutral, while 168 affirmed the effect. 155 of these 168 articles cited Wickström and Bendix as affirming the Hawthorne Effect.”

“Out of 197 affirmative citations of Franke and Kaul, 189 cited the critical articles as affirming the Hawthorne Effect. For Jones, the number was 60 of 103, for Wickström and Bendix 155 of 168…When it comes to academic publishing, the affirming articles are dominant on the issue of the Hawthorne Effect, and are likely the major contributors to the forming of the published consensus.”

“The findings not only demonstrate that the three efforts at criticizing the Hawthorne Effect to varying degrees were unsuccessful, but they also suggest that if the intention behind the critiques were to reduce the frequency of affirmations of the claim in the scientific corpus, they may have achieved the very opposite.”

To read the article, click here.