DATE: Friday 26 October.

PLACE: University of Canterbury, Business School, Meremere, Room 236, Christchurch, NEW ZEALAND

REGISTRATION (important for catering purposes): email to tom.coupe@canterbury.ac.nz

COST: Nada ($0)

Supported by the University of Canterbury Business School Research Committee.

OVERVIEW: There is more and more evidence that findings from many scientific studies cannot be reproduced, casting doubt on the reliability of these studies. At the workshop, we will discuss the extent of this replication crisis, explore various methods that can be used to check whether a study can be replicated, and present tools that can be used to make one’s own research more reproducible and trustworthy.

SPEAKERS AND PRESENTATION TITLES:

– Anton Angelo (University of Canterbury/Library): Transparency and Reproducibility – It’s All About Layers

– Arin Basu (University of Canterbury/Health Sciences): What about Why?

– Annette N. Brown (FHI 360): Which Tests Not Witch Hunts: A Diagnostic Approach to Conducting Replication Research

– Brian Haig (University of Canterbury/Psychology): Understanding Replication in A Way That Is True To Science

– Jeff Miller (University of Otago/Psychology): The Statistical Fundamentals of (Non)-Replicability

– Thomas Pfeiffer (Massey University/Computational Biology/Biochemistry): Betting On Your Peers’ Results: A Tale of Three Markets

– W. Robert Reed (University of Canterbury/Business School): An Update on the Progress of Replications in Economics

– Philip Schluter (University of Canterbury/Health Sciences): A Bayesian Alternative to Hypothesis Testing

– Eric Vanman (University of Queensland/Psychology). How Pre-Registrations Can Improve Science: Tales from the Front-Line

– Ben Wood (Integra LLC): Lessons Learned From Running a Social Science Replication Program

PROGRAMME

9.00-9.30: Registration

9.30-9.35: Introduction

9.35-11.05: SESSION: Replication – Theory and Current Status

11.05-11.30: Coffee Break

11.30-12.30: SESSION: How to Detect the Truth

12.30-13.30: Lunch

13.30-15.00: SESSION: Making Research More Reproducible

15.00-15.30: Coffee Break

15.30-16.30: SESSION: Observations from the Front Lines

16.30-17.00: Closing

ABSTRACTS OF TALKS

SESSION: Replication – Theory and Current Status

– W. Robert Reed (University of Canterbury/Business School): An Update on the Progress of Replications in Economics

Abstract. The last two decades have seen increasing doubt about the credibility of empirical research in science. This has come to be known as the “Replication Crisis,” with the name derived from the fact that many reported empirical findings cannot be reproduced/replicated. Relative to their peers in psychology and political science, economists have been slow to recognize the problem and consider solutions. In 2015, Duvendack et al. published “Replications in Economics: A Progress Report”. Among other things, that study reported (i) the number of replications published in economics over time, (ii) journals that state they publish replications on their websites, (iii) journals that actually publish replications, and (iv) journals that regularly publish data and code along with their empirical papers. This presentation will update those numbers and identify recent trends.

– Jeff Miller (University of Otago/Psychology): The Statistical Fundamentals of (Non)-Replicability

Abstract. A popular conception of science holds that real phenomena should always be replicable when the appropriate conditions are met. Unfortunately, this conception does not hold in scientific fields with inherently probabilistic measurements; in such fields, real phenomena do sometimes fail to replicate. Simple statistical models can illuminate why such errors occur and how their probabilities can be computed, while also clarifying two distinct views of “replication probability”. The models also reveal what quantities are needed to determine the probability of a successful replication. The difficulty of getting good estimates of these quantities makes it hard to determine whether recently observed—and much publicized—replication failures are “as expected” or should instead be regarded as signs of flawed scientific practices.

– Brian Haig (University of Canterbury/Psychology): Understanding Replication in A Way That Is True To Science

Abstract. Successful replication of original research is widely held to be the gold standard by which scientific claims are evaluated. However replication research is said to be rare in the behavioural and biomedical sciences, and recent attempts to replicate published findings in these sciences have reported discouraging results. These two facts have led many to conclude that large tracts of science are in the midst of a replication crisis. As a consequence, it has variously been claimed that the affected disciplines constitute poor science, that their knowledge claims are of doubtful validity, and that much needs to be done to improve their research practices. In this presentation, I challenge a number of widely held assertions that have been made about the nature of replication and its place in science. These challenges are based on underappreciated understandings of good scientific practice.

SESSION: How to Detect the Truth

– Philip Schluter (University of Canterbury/Health Sciences): A Bayesian Alternative to Hypothesis Testing

Abstract. The vast majority of quantitative health research is predicated on Positivism and the Frequentist statistical inference machinery. However, it might be argued that this approach and these methods do not always answer the questions that researcher think are (or want to be) answered. A Bayesian alternative exists – which might have appeal. This talk provides a glimpse into this alternative approach.

– Arin Basu (University of Canterbury/Health Sciences): What about Why?

Abstract. This talk will aim to provide a journey through the world of causal thinking focusing on health sciences. In course of this presentation, I’d like to traverse the different perspectives that have shaped our thinking on what constitutes cause. Starting with Mill’s canons, Sewall Wright’s path analysis, we will examine the role of how Hill’s criteria and Ken Rothman’s notions of necessary and sufficient causal linkages have shaped evidence processes in health and related sciences.

SESSION: Making Research More Reproducible

– Annette N. Brown (FHI 360): Which Tests Not Witch Hunts: A Diagnostic Approach to Conducting Replication Research

Abstract. Replication research can be used to explore original study results that researchers consider questionable, but it should also be a tool for reinforcing the credibility of results that are important to policies and programs. The challenge is to design a replication plan open to both supporting the original findings and uncovering potential problems. Ben Wood and I compiled the examples and lessons from several replication studies to provide researchers with an objective list of checks or tests to consider when planning a replication study. We present tips for diagnostic exercises in four groups: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. We also provide a list of don’ts for how to conduct and report replication research. In this presentation, I will summarize these tips and suggestions using some examples from replication research.

– Anton Angelo (University of Canterbury/Library): Transparency and Reproducibility – It’s All About Layers

Abstract. In order for research to be easily replicated and verified some things need to be ‘baked in’ from the start. This talk looks at the layers required for data and analysis – how to describe data meaningfully, licences required for others to be able to reuse your data (even if it’s just to verify it), and mind-set required to make it successful.

– Eric Vanman (University of Queensland/Psychology). How Pre-Registrations Can Improve Science: Tales from the Front-Line

Abstract. Pre-registration is a declaration of the researchers’ plans prior to the start of a study. It includes stating a priori hypotheses, rules about how many and who the participants are, the procedures to be used, and how the data will be analysed. I will talk about some options for completing a pre-registration of any study, from an undergraduate research thesis to a grant-funded project. I will also review some of the ways that we have encouraged pre- registration in the UQ School of Psychology, which includes promoting open science practices to students and staff.

SESSION: Observations from the Front Lines

– Thomas Pfeiffer (Massey University/Computational Biology/Biochemistry): Betting On Your Peers’ Results: A Tale of Three Markets

Abstract. Prediction markets are popular mechanisms for the forecasting of future events. They have been used in a variety of domains, including sports and politics. In a recent series of studies we used prediction markets in science to have scientists bet on the reliability their peers’ publications. The markets predicted the outcomes of the replications well, suggesting that there is knowledge about the reliability of studies in the research community which can be elicited through market mechanisms. This knowledge is valuable for following the dynamics of hypothesis testing, and can also be employed more broadly for optimizing decision-making in science.

– Ben Wood (Integra LLC): Lessons Learned From Running a Social Science Replication Program

Abstract. Over six years, Ben co-designed and co-managed the replication program for the International Initiative for Impact Evaluation (3ie). He will provide an overview of the program and present lessons learned around terminology standardization, incentivizing processes, refereeing replication research, and requiring push button replication. The talk will intersperse anecdotes from the replication program with results from recent replication- related research. He will also dispel the myth that replication research is unpublishable by highlighting recently published replication studies. The lessons learned will provide valuable feedback to the producers, user, and funders of evidence.