ISO-AHOLA: On Reproducibility and Replication in Psychological and Economic Sciences
[This blog is a summary of a longer treatment of the subject that was published in Frontiers in Psychology in June 2017. To read that article, click here.]
Physicists have asked “why is there something rather than nothing?” They have theorized that it had to do with the formation of an asymmetry between matter and antimatter in the fractions of milliseconds after the Big Bang. Psychologists and economists could ask a similar question, “why do psychological and economic phenomena exist?”
A simple answer is because people exist as psychological and economic entities, therefore so do psychological and economic phenomena. Since “nothingness” is not a phenomenon, there are no logical or philosophical reasons to empirically cast some phenomena into the trash bin of nothingness. Moreover, empirical science does not have the tools to do so because a negative, such as “God does not exist”, cannot ever be proved. This state of affairs is the case because:
(1) The presence of evidence for X does not necessarily mean the absence of evidence for Y. X’s existence is not a precondition for Y’s nonexistence unless they are mutually exclusive effects, which is rare.
(2) Empiricists can never test all of the conditions and groups of people on earth; they cannot even think of all conditions that could give rise to a phenomenon. Further, “there is an infinite number of ideas and ways” to test phenomena, and consequently, “no idea ever achieves the status of final truth” (McFall).
(3) Empiricists do not have perfectly reliable and valid measures.
(4) Human events and behaviors are multi-causal in real life, even in lab experiments. This means that the manipulation of a focal independent variable affects other causal factors. Moreover, researchers cannot control for all the possible confounds or even think of all of them, since “everything is correlated with everything else, more or less”( Meehl); thus, a theoretically established phenomenon under study is never zero.
(5) Humans are fickle and elusive, sometimes unbearably simple and at other times, irreducibly complex in their thinking, and sometimes both at the same time. Thus, the human mind is unreproducible from situation to situation. Unlike those in physics (e.g., speed of light), psychological and economic phenomena are not fixed constants in space and time. There are no cognitive dissonance particles or “Phillips curve” particles that could irrevocably be verified by empirical data and subsequently declared universal constants.
In short, the nonexistence of phenomena is not logically viable. Since there are infinite ways of measuring and studying a phenomenon, logically, it should be possible to devise experiments both to demonstrate that a phenomenon exists and that it does not exist in specific conditions and during specific times when using specific methods and specific tasks. The former finding means that the phenomenon has been demonstrated to exist and it cannot be retracted.
On the one hand, conceptual replications can establish a phenomenon’s boundary conditions (i.e., when it is more and less likely to occur). On the other hand, the finding of nonexistence would not invalidate the phenomenon — only that it is not strong enough to register in a specific condition. For example, psychologically, people do not “choke” under all stressful conditions because they have learned to deal with pressure. Similarly, if the Phillips curve does not explain the inverse relationship between unemployment and inflation in the present low-interest economic environment particularly well, it can do so under different economic circumstances.
Does the conditional nature of these phenomena mean they should disappear into a “black hole” of nonexistent phenomena. Of course, not. The best that can be done is to empirically test and conceptually replicate well-developed theories (their tenets) with the best tools available, but never claim that the phenomena they describe and explain do not exist.
The present emphasis on reproducibility in psychological and economic science, unfortunately, stems from the application of the physics model of independent verification of precise numeric values for phenomena, such as three recent independent confirmations of “gravity waves” predicted by Einstein’s theory. This type of replication is possible only if there are universal constants to be verified. However, there are none in psychology and economics, and not even in biology.
Physics seeks to discover the laws of nature, whereas in other sciences, both nature and nurture have to be taken into account. This Person-Environment interaction in human cognition, performance and behavior makes direct and precise replications impossible. Human conditions vary, for one thing, because individuals (investors, policy-makers and politicians) are not invariant and rational in their decisions and judgments. “Behavioral economists” (e.g., Thaler and Kahneman) have shown that the rational-agent model is a poor explanation for financial judgments and decisions, or economic growth more generally. People may consider all the information provided but they are also influenced by their self-generated and environmentally-induced emotions in their judgments and behaviors. Individual investors (e.g., prospective homeowners) can also be led to make boneheaded decisions, resulting in “collective blindness” (Kahneman) that can in turn create national and international financial crises, as was seen in the 2008 financial calamity.
All of this means that there will be deviations from overall patterns of individual financial behaviors, and therefore direct and precise replications are impossible. However, conceptual or constructive replications are helpful in elucidating the boundary conditions for the overall pattern, conditions under which a phenomenon is strong and weak. But if we insist on precise replications, then no psychological or economic phenomena exist because it impossible to have and create identical conditions to those of the original testing.
There are no universal constants to be precisely replicated outside the laws of nature and physics. If the conditions are not the same at the individual level, they are not the same at the macro level either. History does not exactly repeat itself, it only rhymes (Twain). The conditions that led to a recession at one time will not be the same causes for the next recession. At the macro level, researchers can build theoretical models trying to predict the next recession, but they conceivably cannot consider all relevant variables, especially exogenous ones, and thus precise predictions (replications) are not possible. Nevertheless, this does not prevent pundits from arguing that it is “different this time”, it is “a new normal”.
A replication’s success is typically determined by statistical means (traditionally p-value and now Effect Size). But psychological and economic phenomena cannot be reduced to statistical phenomena, and theoretical and methodological deficiencies cannot be saved by statistical analyses. Science mainly advances by theory building and model construction, not by empirical testing and replication of the statistical null hypothesis. Psychological and economic phenomena are largely theoretical constructs, not unlike those in physics. Just think where physics would be today without Einstein’s theories. The Higgs boson particle was theorized to exist in 1964 but not verified until 2012. Did the particle not exist in the meantime?
Thus, empirical studies are mainly evaluated for their theoretical relevance and importance, and less for their success or failure in exactly reproducing original findings. It is not empirical data but theory that has generally made scientific progress possible, which is as true of physics as it is of psychology and economics. Along the way, empirical data have complemented and contributed to the expansion of theoretical models, and theories have made data more useful. Of course, theories are eventually abandoned in Kuhnian-like paradigm shifts. In the meantime, there are only “temporary winners” as scientific knowledge is “provisional” and “propositional” in nature.
Seppo Iso-Ahola is Professor of Psychology in Kinesiology at the School of Public Health, University of Maryland. He can be contacted at email@example.com.
*The author thanks Roger C. Mannell for his helpful comments and suggestions.