The International Journal for Re-Views in Empirical Economics (IREE) is the only journal in economics solely dedicated to publishing replications. Recently, IREE was evaluated by TOP Factor. TOP Factor is an initiative launched by the Center for Open Science to assess journals according to “a values-aligned rating of journal policies as a counterweight to metrics that incentivize journals to publish exciting results regardless of credibility” (see here). The assessment of IREE‘s journal policies resulted in a journal score of 13 points. This puts IREE in 4th place among all 136 economic journals rated by TOP Factor, ahead of the American Economic Review, Econometrica, Plos One, and Science.
TOP Factor provides an alternative to metrics such as the journal impact factor (JIF). It constitutes a first step towards evaluating journals based on their quality of process and implementation of scholarly values. “Too often, journals are compared using metrics that have nothing to do with their quality,” says Evan Mayo-Wilson, Associate Professor in the Department of Epidemiology and Biostatistics at Indiana University School of Public Health-Bloomington. “The TOP Factor measures something that matters. It compares journals based on whether they require transparency and methods that help reveal the credibility of research findings.” (see COS announcement of TOP factor, 2020).
TOP Factor is based on the Transparency and Openness Promotion (TOP) Guidelines, a framework of eight standards that summarize behaviors that can improve transparency and reproducibility of research such as transparency of data, materials, code, and research design, preregistration, and replication.
Editor Martina Grunow announced that she was very pleased with this rating, as TOP Factor reflects exactly what IREE stands for: reducing the publication bias towards literally incredible and non-reproducible results and the resulting “publish-or-perish” culture. Like TOP Factor, IREE promotes the reproducibility and transparency of published results and scientific discourse in economics based on high-quality and credible research results.
[Excerpts are taken from the blog “Evidence of Fraud in an Influential Field Experiment About Dishonesty” posted by Uri Simonsohn, Joe Simmons, Leif Nelson and anonymous researchers at Data Colada]
“This post is co-authored with a team of researchers who have chosen to remain anonymous. They uncovered most of the evidence reported in this post.”
“In 2012, Shu, Mazar, Gino, Ariely, and Bazerman published a three-study paper in PNAS reporting that dishonesty can be reduced by asking people to sign a statement of honest intent before providing information (i.e., at the top of a document) rather than after providing information (i.e., at the bottom of a document).”
“In 2020, Kristal, Whillans, and the five original authors published a follow-up in PNAS entitled, “Signing at the beginning versus at the end does not decrease dishonesty”.
“Our focus here is on Study 3 in the 2012 paper, a field experiment (N = 13,488) conducted by an auto insurance company … under the supervision of the fourth author. Customers were asked to report the current odometer reading of up to four cars covered by their policy.”
“The authors of the 2020 paper did not attempt to replicate that field experiment, but they did discover an anomaly in the data…our story really starts from here, thanks to the authors of the 2020 paper, who posted the data of their replication attempts and the data from the original 2012 paper.”
“A team of anonymous researchers downloaded it, and discovered … very strong evidence that the data were fabricated.”
“Let’s start by describing the data file. Below is a screenshot of the first 12 observations:”
“You can see variables representing the experimental condition, a masked policy number, and two sets of mileages for up to four cars. The “baseline_car[x]” columns contain the mileage that had been previously reported for the vehicle x (at Time 1), and the “update_car[x]” columns show the mileage reported on the form that was used in this experiment (at Time 2).”
“On to the anomalies.”
Anomaly #1: Implausible Distribution of Miles Driven
“Let’s first think about what the distribution of miles driven should look like…we might expect…some people drive a whole lot, some people drive very little, and most people drive a moderate amount.”
“As noted by the authors of the 2012 paper, it is unknown how much time elapsed between the baseline period (Time 1) and their experiment (Time 2), and it was reportedly different for different customers. … It is therefore hard to know what the distribution of miles driven should look like in those data.”
“It is not hard, however, to know what it should not look like. It should not look like this:”
“First, it is visually and statistically (p=.84) indistinguishable from a uniform distribution ranging from 0 miles to 50,000 miles. Think about what that means. Between Time 1 and Time 2, just as many people drove 40,000 miles as drove 20,000 as drove 10,000 as drove 1,000 as drove 500 miles, etc. This is not what real data look like, and we can’t think of a plausible benign explanation for it.”
“Second, there is some weird stuff happening with rounding…”
Anomaly #2: No Rounded Mileages At Time 2
“The mileages reported in this experiment … are what people wrote down on a piece of paper. And when real people report large numbers by hand, they tend to round them.”
“Of course, in this case some customers may have looked at their odometer and reported exactly what it displayed. But undoubtedly many would have ballparked it and reported a round number.”
“In fact, as we are about to show you, in the baseline (Time 1) data, there are lots of rounded values.”
“But random number generators don’t round. And so if, as we suspect, the experimental (Time 2) data were generated with the aid of a random number generator (like RANDBETWEEN(0,50000)), the Time 2 mileage data would not be rounded.”
“The figure shows that while multiples of 1,000 and 100 were disproportionately common in the Time 1 data, they weren’t more common than other numbers in the Time 2 data.”
“These data are consistent with the hypothesis that a random number generator was used to create the Time 2 data.”
“In the next section we will see that even the Time 1 data were tampered with.”
Interlude: Calibri and Cambria
“Perhaps the most peculiar feature of the dataset is the fact that the baseline data for Car #1 in the posted Excel file appears in two different fonts. Specifically, half of the data in that column are printed in Calibri, and half are printed in Cambria.”
“The analyses we have performed on these two fonts provide evidence of a rather specific form of data tampering.”
“We believe the dataset began with the observations in Calibri font. Those were then duplicated using Cambria font. In that process, a random number from 0 to 1,000 (e.g., RANDBETWEEN(0,1000)) was added to the baseline (Time 1) mileage of each car, perhaps to mask the duplication.”
“In the next two sections, we review the evidence for this particular form of data tampering.”
Anomaly #3: Near-Duplicate Calibri and Cambria Observations
“…the baseline mileages for Car #1 appear in Calibri font for 6,744 customers in the dataset and Cambria font for 6,744 customers in the dataset. So exactly half are in one font, and half are in the other. For the other three cars, there is an odd number of observations, such that the split between Cambria and Calibri is off by exactly one (e.g., there are 2,825 Calibri rows and 2,824 Cambria rows for Car #2).”
“… each observation in Calibri tends to match an observation in Cambria.”
“To understand what we mean by “match” take a look at these two customers:”
“The top customer has a “baseline_car1” mileage written in Calibri, whereas the bottom’s is written in Cambria. For all four cars, these two customers have extremely similar baseline mileages.”
“Indeed, in all four cases, the Cambria’s baseline mileage is (1) greater than the Calibri mileage, and (2) within 1,000 miles of the Calibri mileage. Before the experiment, these two customers were like driving twins.”
“Obviously, if this were the only pair of driving twins in a dataset of more than 13,000 observations, it would not be worth commenting on. But it is not the only pair.”
“There are 22 four-car Calibri customers in the dataset. All of them have a Cambria driving twin…there are twins throughout the data, and you can easily identify them for three-car, two-car, and unusual one-car customers, too.”
“To see a fuller picture of just how similar these Calibri and Cambria customers are, take a look at Figure 5, which shows the cumulative distributions of baseline miles for Car #1 and Car #4.”
“Within each panel, there are two lines, one for the Calibri distribution and one for the Cambria distribution. The lines are so on top of each other that it is easy to miss the fact that there are two of them:”
Anomaly #4: No Rounding in Cambria Observations
“As mentioned above, we believe that a random number between 0 and 1,000 was added to the Calibri baseline mileages to generate the Cambria baseline mileages. And as we have seen before, this process would predict that the Calibri mileages are rounded, but that the Cambria mileages are not.”
“This is indeed what we observe:”
“The evidence presented in this post indicates that the data underwent at least two forms of fabrication: (1) many Time 1 data points were duplicated and then slightly altered (using a random number generator) to create additional observations, and (2) all of the Time 2 data were created using a random number generator that capped miles driven, the key dependent variable, at 50,000 miles.”
“We have worked on enough fraud cases in the last decade to know that scientific fraud is more common than is convenient to believe… There will never be a perfect solution, but there is an obvious step to take: Data should be posted.”
“The fabrication in this paper was discovered because the data were posted. If more data were posted, fraud would be easier to catch. And if fraud is easier to catch, some potential fraudsters may be more reluctant to do it. … All of our journals should require data posting.”
“Until that day comes, all of us have a role to play. As authors (and co-authors), we should always make all of our data publicly available. And as editors and reviewers, we can ask for data during the review process, or turn down requests to review papers that do not make their data available.”
“A field that ignores the problem of fraud, or pretends that it does not exist, risks losing its credibility. And deservedly so.”
To read the full blog, click here.
[Excerpts are taken from the article “Retracted: Risk Management in Financial Institutions” “ by Adriano Rampini, S. Viswanathan, and Guillaume Vuillemey, published in the Journal of Finance]
“The authors hereby retract the above article, published in print in the April 2020 issue of The Journal of Finance. A replication study finds that the replication code provided in the supplementary information section of the article does not reproduce some of the central findings reported in the article.”
“Upon reexamination of the work, the authors confirmed that the replication code does not fully reproduce the published results and were unable to provide revised code that does. Therefore, the authors conclude that the published results are not reliable and that the responsible course of action is to retract the article and return the Brattle Group Distinguished Paper Prize that the article received.”
InSPiR2eS is a new global research network primarily aimed at research training and capacity building, resting on a foundation theme of responsible science (for some more details, please refer to the 2-pager outline here).
Whether you are a current network member or not, you are warmly invited to the 1-hour webinar launch of the network taking place during the window, 22-24th June.
For your convenience, the Zoom launch is offered in 3 separate repeat events summarised below (please see here for a doc that gives more details confirming equivalent dates/times for your part of the world):
#1: Tuesday 22nd June at 18:00 Australian Eastern Standard Time (AEST)
Topic: Robert Faff’s Zoom Meeting #1 launching InSPiR2eS
Join from a PC, Mac, iOS or Android: https://bond.zoom.us/j/91798395671
#2: Wednesday 23rd June at 15:00 AEST
Topic: Robert Faff’s Zoom Meeting #2 launching InSPiR2eS
Join from a PC, Mac, iOS or Android: https://bond.zoom.us/j/99592757933
#3: Thursday 24th June at 06:00 AEST
Topic: Robert Faff’s Zoom Meeting #3 launching InSPiR2eS
Join from a PC, Mac, iOS or Android: https://bond.zoom.us/j/92796833442
If you are interested in joining the Zoom launch of InSPiR2eS, please register ASAP at the Google Docs link here.
Finally, please share this open invitation with whomever you think might be interested. Thank you!
What is the International Society of Pitching Research for Responsible Science (InSPiR2eS) research network?
InSPiR2eS is a globally-facing research network primarily aimed at research training and capacity building, resting on a foundation theme of responsible science.
As measures of its success, the network succeeds if (beyond what we would have otherwise achieved) it inspires:
– responsible research – i.e., research that produces new knowledge that is credible, useful & independent.
– productive research collaboration & partnerships – locally, regionally & globally.
– a collective sense of purpose and achievement towards the whole research process.
Importantly, the network aims to inclusively embrace like-minded university researchers centred on the multi-faceted utility provided by the “Pitching Research” framework, as a natural enabler of responsible science.
The network alliance is a fully “opt in” organisational structure. Through the very act of joining the network, each member will abide by an appropriate Code of Conduct (under development), including: privacy, confidentiality, communication and notional IP relating to research ideas.
Why create InSPiR2eS?
The underlying premise for creating InSPiR2eS is to facilitate an efficient co-ordinated sharing of relevant research information and resources for the mutual benefit of all participants – whether this occurs through the inputs, processes or outputs linked to our research endeavours. More generally, while the enabling focus is on the Pitching Research framework, the network can offer its members a global outreach for their research efforts – in new and novel ways.
For example, actively engaging the network could spawn new research teams and projects, or other international initiatives and alliances. While such positive outcomes could “just happen” anyway, absent creating a new network – which is hardly novel, the network could e.g., experiment with a “shark tank” type webinar event in which some members pitch for new project collaborators. Exploiting the power of a strong alliance, the network can deliver highly leveraged outcomes compared to what is possible when we act alone – as isolated “sole traders”.
Who are the members of InSPiR2eS?
Professor Robert Faff (Bond University), as the network initiator, is the network convenor and President of InSPiR2eS. Currently, the network has more than 500 founding Ambassadors, Members and Associate Members already signed up representing 73 countries/ jurisdictions: Australia, Pakistan, China, Canada, New Zealand, Vietnam, Brazil, Nigeria, Germany, Indonesia, the Netherlands, England, Kenya, Romania, Poland, Mauritius, Sri Lanka, Bangladesh, Italy, Spain, India, Scotland, Singapore, Japan; Norway; Ireland; the US; Malaysia; Chile; Turkey; Wales; Serbia; Belgium; Thailand; France; South Africa; Switzerland; Croatia; Czech Republic; Hong Kong; Taiwan; Macau; South Korea; Greece; Ukraine; Ghana; Slovenia; Austria; Cyprus; Uganda; Namibia; Portugal; Tanzania; Fiji; Saudi Arabia; Estonia; Iceland; Egypt; Mongolia; Lithuania; Slovakia; Finland; Sweden; Ecuador; Israel; Hungary; UAE; North Cyprus; Mozambique; Philippines; Nepal; Argentina; Malta.
How will InSPiR2eS operate?
Phase 1: Network setup and initial information exchange.
To begin with, we will rely (mostly) on email communication. We will establish an e-newsletter – to provide engaging and organised information exchange. Dr Searat Ali (University of Wollongong) has agreed to be the inaugural Editor of the InSPiR2eS e-Newsletter (in his role as VP – Communications).
Phase 2: Establishing interactive network engagement.
Live webinar Zoom sessions will be offered on topics linked to the network Mission. These sessions would be recorded and freely accessible from an InSPiR2eS “Resource Library”. Initially, these sessions will be presented by the network leader, but over time others in the network would be welcome to offer sessions – especially, if the topics are of a general nature aiming for research training/capacity building (rather than a research seminar on their latest paper). These webinars would be open to all, irrespective of whether they are network members or not – including network members, as well as to their students, their research collaborators and any other invited associated researcher in their networks.
The inaugural network webinar will broadly address the core theme of “responsible science”, and this material will serve as a beacon against which all network activities will be offered. Subsequent webinar topics might include the following modules:
– A Basic Primer on Pitching Research.
– Using Pitching Research as a Reverse Engineering Tool.
– Advanced Guidelines on applying the Pitching Research Framework.
– Pitching Research for Engagement & Impact.
– Pitching Research as a Tool for Responsible Science.
– Pitching Research as a Tool for Replications.
– Pitching Research as a Tool for Pre-registration.
– Pitching Research for Diagnostic use in Writing.
– Roundtable Panels – e.g., discussing issues related to “responsible science”, etc.
Phase 3: Longer-term, post-COVID network initiatives.
Downstream network initiatives will include the creation of a “one-stop shop” network website. And, once COVID is behind us, we will explore some in-person events like:
– a conference or symposium.
– “shark tank” event(s), either themed on “pitching research for finding collaborators” or “pitching research to journal editors”.
– initiatives/ special projects/ network events suggested and/or co-ordinated by network members.
When will InSPiR2eS content activity begin?
Release of the inaugural edition of the e-Newsletter will be a signature activity and we are aiming for this to be ready later in May, 2021. Zoom webinars, will also start soon – we are aiming for a network opening event in June 2021. Please keep an eye for publicity on this event soon.
How do I join the InSPiR2eS research network?
If you are interested in joining the InSPiR2eS research network and engaging in its upcoming rich program of webinars, workshops and research resources, then register at the following Google Docs link (click here). In locations where Google is problematic, click here.
Robert Faff is Professor of Finance at Bond University. He is Network Convenor & President of InSPiR2eS. Professor Faff can be contacted at firstname.lastname@example.org.
 In part, the idea of the network itself is inspired by the community for Responsible Research in Business and Management that released a position paper in 2017, in which they outline a vision for the year 2030 “… of a future in which business schools and scholars worldwide have successfully transformed their research toward responsible science, producing useful and credible knowledge that addresses problems important to business and society.”