Inter-American Development Bank
facebook
twitter
youtube
linkedin
instagram
Abierto al públicoBeyond BordersCaribbean Development TrendsCiudades SosteniblesEnergía para el FuturoEnfoque EducaciónFactor TrabajoGente SaludableGestión fiscalGobernarteIdeas MatterIdeas que CuentanIdeaçãoImpactoIndustrias CreativasLa Maleta AbiertaMoviliblogMás Allá de las FronterasNegocios SosteniblesPrimeros PasosPuntos sobre la iSeguridad CiudadanaSostenibilidadVolvamos a la fuente¿Y si hablamos de igualdad?Home
Citizen Security and Justice Creative Industries Development Effectiveness Early Childhood Development Education Energy Envirnment. Climate Change and Safeguards Fiscal policy and management Gender and Diversity Health Labor and pensions Open Knowledge Public management Science, Technology and Innovation  Trade and Regional Integration Urban Development and Housing Water and Sanitation
  • Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer

Impacto

  • HOME
  • CATEGORIES
    • Beyond development effectiveness
    • Evaluation methods and techniques
    • Measuring our performance
    • What does and doesn’t work in development
  • Authors
  • English
    • Español

Is economics research credible?

December 20, 2011 por Autor invitado Leave a Comment


By Conner Mullally

Many of us believe that studies following the “gold standard” of randomized assignment, such as those generated by academic labs in the hard sciences, are beyond reproach.

According to Bruce Booth – a scientist turned venture capitalist – “the unspoken rule is that at least 50% of the studies published even in top tier academic journals – Science, Nature, Cell, PNAS, etc… – can’t be repeated with the same conclusions by an industrial lab.” This is an absolutely terrifying quote for development researchers and consumers of research products alike.

Recently, the economics blogosphere has taken up the issue of research credibility, specifically with regard to how much we ought to trust the results of statistical hypothesis testing in economic research.

Excellent discussions include recent posts by Alex Tabarrok at Marginal Revolution and Ole Rasmussen for the World Bank Development Research Group blog.

Both offer suggestions for improving the degree to which we can trust in the results of empirical work in development, drawing upon the literature on the subject inside and outside of economics.

Any hypothesis, no matter how absurd, always has a positive probability of not being rejected. Thus if enough scholars are working on testing a particular hypothesis, there will always exist studies that fail to reject it, even though it is false.

True hypotheses face the same problem in that there would always be studies that reject them, despite being true. This is not a problem if one considers the literature as a whole, e.g. through meta-analyses or review papers.

Then the probabilities of false rejection or mistaken acceptance of hypotheses ought to show up in the estimated distribution generated by different studies.

But journals and scholars alike have a mutually reinforcing preference for studies with statistically significant results; e.g., see de Long and Lang (1992). As a result, meta-analyses and review papers will suffer from selection bias, so the literature will not be representative of actual findings.

The credibility of peer-reviewed research is foundational to the soundness of a science as a whole. Being published is also what largely determines upward mobility in the academic world. But it is far from clear that scholarly journals are at the center of policy debates, and it is policy that is of interest to IDB and development practitioners more generally.

Most studies are around for years in working paper form prior to being published in academic journals, and many of these studies will never be published at all. Studies from working paper series hosted by institutions such as IDB, the World Bank, the National Bureau of Economic Research, and others may therefore hold more sway in current policy debates than peer-reviewed articles; the latter are “old news” in a sense.

This raises three interesting questions.

Firstly, what happens to studies that are not published (or even submitted for publication) because of imprecise estimates of impacts?

Second, are they doomed to languish on hard drives, or thirdly, and finally, do they exist in working paper form?  If it is the latter, then this suggests that there exists a tradeoff between the unfiltered nature of working papers and the screening process offered by the peer-review system. Which body of literature is more affected by bias that takes us further away from the truth would remain an open question.

Conner Mullally is a consultant at the Office of Strategic Planning and Development Effectiveness at the IDB. He is originally from Seattle, WA and received his PhD in agricultural and resource economics from the University of California, Davis this year.


Filed Under: Beyond development effectiveness Tagged With: absurd hypothesis, Alex Tabarrok, Bruce Booth, false rejection, gold standard, J. Bradford de Long, Kevin Lang, Marginal Revolution, meta-analyses, mistaken acceptance, Ole Rasmussen, randomized assignment, research credibility, selection bias, true hypotheses

Autor invitado

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Follow Us

Subscribe

Search

About this blog

This blog highlights effective ideas in the fight against poverty and exclusion, and analyzes the impact of development projects in Latin America and the Caribbean.

Categories

Footer

Banco Interamericano de Desarrollo
facebook
twitter
youtube
youtube
youtube

    Blog posts written by Bank employees:

    Copyright © Inter-American Development Bank ("IDB"). This work is licensed under a Creative Commons IGO 3.0 Attribution-NonCommercial-NoDerivatives. (CC-IGO 3.0 BY-NC-ND) license and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC- IGO license. Note that link provided above includes additional terms and conditions of the license.


    For blogs written by external parties:

    For questions concerning copyright for authors that are not IADB employees please complete the contact form for this blog.

    The opinions expressed in this blog are those of the authors and do not necessarily reflect the views of the IDB, its Board of Directors, or the countries they represent.

    Attribution: in addition to giving attribution to the respective author and copyright owner, as appropriate, we would appreciate if you could include a link that remits back the IDB Blogs website.



    Privacy Policy

    Derechos de autor © 2025 · Magazine Pro en Genesis Framework · WordPress · Log in

    Banco Interamericano de Desarrollo

    Aviso Legal

    Las opiniones expresadas en estos blogs son las de los autores y no necesariamente reflejan las opiniones del Banco Interamericano de Desarrollo, sus directivas, la Asamblea de Gobernadores o sus países miembros.

    facebook
    twitter
    youtube
    This site uses cookies to optimize functionality and give you the best possible experience. If you continue to navigate this website beyond this page, cookies will be placed on your browser.
    To learn more about cookies, click here
    X
    Manage consent

    Privacy Overview

    This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
    Necessary
    Always Enabled
    Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
    Non-necessary
    Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
    SAVE & ACCEPT