QUICK: Does finding a significant effect when the sample size is small make it more likely that the effects are real and important?  Or less?

James Heckman, Nobel Prize winning economist, says more:

“Also holding back progress are those who claim that Perry and ABC are experiments with samples too small to accurately predict widespread impact and return on investment. This is a nonsensical argument. Their relatively small sample sizes actually speak for — not against — the strength of their findings. Dramatic differences between treatment and control-group outcomes are usually not found in small sample experiments, yet the differences in Perry and ABC are big and consistent in rigorous analyses of these data.” (click here for source).

Andrew Gelman, Professor Statistics and Political Science at Columbia University, and blogger extraordinaire, says less:

“I agree with Stuart Buck that Heckman is wrong here. Actually, the smaller sample sizes (and also the high variation in these studies) speaks against—not for—the strength of the published claims.”

Who do YOU think is right?

To read more from Gelman, click here. His argument is elaborated in this working paper.