It is common for authors of empirical studies to use the conclusion of their paper to summarize their empirical findings, without explicitly discussing why their results might differ from previous studies, nor suggesting ways to resolve observed discrepancies. In a recent blog, Arnaud Vaganay suggests the following, and while it is targeted towards replications, it applies to any empirical study that addresses a topic that has been studied by previous literature.

“A reproducible discussion includes two main steps. The first step is the systematic comparison of your results with results from previous studies (as mentioned above). Inasmuch as possible, results should be compared head-to-head using both: (i) Unstandardized values: by comparing the direction and statistical significance of your results with the same quantities in previous studies; (ii) Standardized values: by comparing the magnitude/size of your effect with the magnitude/size of effects in previous studies. Ideally, an additional test should assess whether the difference between these effects is statically significant.”

“If your results cannot be directly compared (for example because your study analysed the data in a novel way), you should clearly mention it and invite further replications. As previously mentioned, it is through replication that the credibility of a theory can be ascertained.”

“The second step consists in correctly interpreting findings. If your results are in line with previous results, the effect is robust and the theory is corroborated (assuming no p-hacking of course). If the results are significantly different, the plausibility of the following scenarios should be discussed: (i) Your study differs significantly in terms of analysis …; (ii) Your study differs significantly in terms of intervention/independent variable … ; (iii) Your study differs significantly in terms of sample; (iv) Your study differs significantly in terms of social, cultural or institutional context.”

“Ideally: (i) These hypotheses should be tested in subsequent studies; (ii) These two steps should be pre-registered and any change to the original protocol flagged and justified.”

To read more, click here.