By Conner Mullally
Many of us believe that studies following the “gold standard” of randomized assignment, such as those generated by academic labs in the hard sciences, are beyond reproach.
According to Bruce Booth – a scientist turned venture capitalist – “the unspoken rule is that at least 50% of the studies published even in top tier academic journals – Science, Nature, Cell, PNAS, etc… – can’t be repeated with the same conclusions by an industrial lab.” This is an absolutely terrifying quote for development researchers and consumers of research products alike.
Recently, the economics blogosphere has taken up the issue of research credibility, specifically with regard to how much we ought to trust the results of statistical hypothesis testing in economic research.
Both offer suggestions for improving the degree to which we can trust in the results of empirical work in development, drawing upon the literature on the subject inside and outside of economics.
Any hypothesis, no matter how absurd, always has a positive probability of not being rejected. Thus if enough scholars are working on testing a particular hypothesis, there will always exist studies that fail to reject it, even though it is false.
True hypotheses face the same problem in that there would always be studies that reject them, despite being true. This is not a problem if one considers the literature as a whole, e.g. through meta-analyses or review papers.
Then the probabilities of false rejection or mistaken acceptance of hypotheses ought to show up in the estimated distribution generated by different studies.
But journals and scholars alike have a mutually reinforcing preference for studies with statistically significant results; e.g., see de Long and Lang (1992). As a result, meta-analyses and review papers will suffer from selection bias, so the literature will not be representative of actual findings.
The credibility of peer-reviewed research is foundational to the soundness of a science as a whole. Being published is also what largely determines upward mobility in the academic world. But it is far from clear that scholarly journals are at the center of policy debates, and it is policy that is of interest to IDB and development practitioners more generally.
Most studies are around for years in working paper form prior to being published in academic journals, and many of these studies will never be published at all. Studies from working paper series hosted by institutions such as IDB, the World Bank, the National Bureau of Economic Research, and others may therefore hold more sway in current policy debates than peer-reviewed articles; the latter are “old news” in a sense.
This raises three interesting questions.
Firstly, what happens to studies that are not published (or even submitted for publication) because of imprecise estimates of impacts?
Second, are they doomed to languish on hard drives, or thirdly, and finally, do they exist in working paper form? If it is the latter, then this suggests that there exists a tradeoff between the unfiltered nature of working papers and the screening process offered by the peer-review system. Which body of literature is more affected by bias that takes us further away from the truth would remain an open question.
Conner Mullally is a consultant at the Office of Strategic Planning and Development Effectiveness at the IDB. He is originally from Seattle, WA and received his PhD in agricultural and resource economics from the University of California, Davis this year.