Microfinance is one of the most hotly debated interventions in developing countries over the last 20 years. These are generally small loans, often given to women with short repayment periods and high interest rates (though often much lower than local market rates).

Proponents argue that the poor often are severely cash constrained and a little bit of money can help them to realize their economic potential. Researchers and policy makers have been worried about people being caught in debt traps, making their poverty worse. Prior to the rise of randomized control trials (RCTs) in economics the evidence either way was mostly anecdotal.

Then some experimental evidence started to emerge, and the results were not encouraging. Karlan and Zinman (2011) and then six other RCT studies published in a 2015 special issue of the American Economic Journal: Applied Economics found no statistically significant economic impacts from giving people small loans. In Fiala (2018) I found some positive short-run impacts, but only for men.

However, not all evidence, even experimental, is created equal. In Dahal and Fiala (2020) we closely re-analyze these eight papers to examine just how well these studies were designed to answer the questions they wanted to answer. We find that the lack of statistical significance is likely not due to a lack of impacts. Rather, the problem is that these studies are extremely underpowered. I

ndividual coefficients are actually quite large, but the standard errors are even bigger. Ex-post power calculations for each of the studies show the minimum detectable effect (MDE) size for main outcomes is up to 1,000%. Median (mean) MDE is 132% (201%). The authors find effects closer to 30%, a large impact but far from what is needed to be statistically significant.

Why are these studies so underpowered? One of the biggest reasons is that there is significant non-compliance. Take-up rates of loans in the treatment groups is generally low. Often the difference between take-up of loans between treatment and control groups is tiny. Three of the American Economic Journal: Applied Economics papers have net compliance rates less than 10 percentage points, making reasonable inference almost impossible.

While some of the authors of the original studies acknowledge potential issues of low power, they never quantify them. This lack of transparency has led many people, including the original authors, to describe the null results as precisely estimated.

Our analysis opens up a bigger problem within experimental methods in general: RCTs can be the gold standard of inference, but only when designed and implemented properly on questions they can be used to answer.

The problems of design in microfinance isn’t due to low quality researchers running poor quality studies. Two of the recent Noble Prize winners in Economics are authors of these studies, and one of them was the editor of the American Economic Journal: Applied Economics when these studies were published. One of the papers had been a working paper for years. Even looking back at the standards of 2015, none of these studies passed what was considered appropriate quality.

Although these papers do not show a “transformative” impact of microfinance on the lives of poor households, careful reading of the papers reveals that they also do not discredit the role of microcredit in poverty alleviation and improving livelihoods of poor households.

The main conclusion of Dahal and Fiala (2020) is that we actually have no idea about the impact of microfinance on the lives of poor people because there is not a single study looking at the impact of the traditional microfinance model that is designed well enough to answer this question.

Nathan Fiala is an assistant professor at the University of Connecticut, honorary senior lecturer at Makerere University in Uganda, and a research fellow at RWI in Essen, Germany. He can be contacted at nathan.fiala@uconn.edu.