Don’t Have Time To Do a Replication? Have You Considered p-Curves?
So another study finds that X affects Y, and you are a sufficiently cynical TRN reader that you wonder if the authors have p-hacked their way to get their result. Don’t have time (or the incentive) to do a replication? You might consider using a “p-curve” analysis to determine whether the effect has “evidentiary value.” How does one do that? Let’s take as a given that most journals will not publish a result unless it is statisically significant. Even if the journals only report significant results, one can examine the distribution of p-values to determine whether or not the effect is true. Want to learn more about “p-curves?” The original article by Simonsohn, Nelson, & Simmons (2011), “P-Curve: A Key to the File Drawer” can be found here. A straightforward explanation of the technique by Will Gervais can be found here. A critique by Bruns & Ioannidis can be found here. And an excellent response by the original authors can be found here. P-curves are definitely worth a look!
Like this:
Like Loading...
The Replication Network 
Note that the conversation between Simonsohn et al. and Bruns and Ioannides (B&I) continues with this response here: http://datacolada.org/wp-content/uploads/2016/06/Reply_to_SNS_160608.pdf .
LikeLike