Inter-American Development Bank
facebook
twitter
youtube
linkedin
instagram
Abierto al públicoBeyond BordersCaribbean Development TrendsCiudades SosteniblesEnergía para el FuturoEnfoque EducaciónFactor TrabajoGente SaludableGestión fiscalGobernarteIdeas MatterIdeas que CuentanIdeaçãoImpactoIndustrias CreativasLa Maleta AbiertaMoviliblogMás Allá de las FronterasNegocios SosteniblesPrimeros PasosPuntos sobre la iSeguridad CiudadanaSostenibilidadVolvamos a la fuente¿Y si hablamos de igualdad?Home
Citizen Security and Justice Creative Industries Development Effectiveness Early Childhood Development Education Energy Envirnment. Climate Change and Safeguards Fiscal policy and management Gender and Diversity Health Labor and pensions Open Knowledge Public management Science, Technology and Innovation  Trade and Regional Integration Urban Development and Housing Water and Sanitation
  • Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer

Impacto

  • HOME
  • CATEGORIES
    • Beyond development effectiveness
    • Evaluation methods and techniques
    • Measuring our performance
    • What does and doesn’t work in development
  • Authors
  • English
    • Español

Evaluating the impact of science, technology and innovation programs: a toolkit

February 22, 2012 por Alessandro Maffioli Leave a Comment


Evaluating the impact of science

As mentioned in a previous blog by my colleague Pablo Ibarrarán, the Office of Strategic Planning and Development Effectiveness of the IDB is publishing impact evaluation guidelines on selected topics and methods.

The last publication of this series – Evaluating the Impact of Science, Technology and Innovation Programs: a Methodological (STIP) Toolkit, co-authored by myself, Gustavo Crespi, Pierre Monhen and Gonzalo Vázquez – reflects a joint effort of our office and the recently created Competitiveness and Innovation Division at the IDB to push the agenda of evidence-based policy making in the area of Production Development Policies.

When we conceptualized and developed this toolkit our main objective was to facilitate the dialogue between STIP specialists and impact evaluation experts, by providing the former a complete and simple review of evaluation methods and the latter some key highlights of the specific challenges of evaluating STIP.

For this purpose, we first discuss in detail the theory of change at the basis of the STIP and indentify the fundamental evaluation questions and the typical outcome of interests. We then devote a great deal of attention to the topic of data: we discuss pros and cons of different data sources, data quality issues, and strategies for data collection.

In what is probably the core of the paper, we analyze in detail the potential application of experimental and quasi-experimental methods to STIP.

For each method, we highlight characteristics and assumptions, practical issues related to its implementation, and strengths and weakness specifically related to the application to STIP.

In an effort to be as practical as possible, we refer to many examples of existing and ongoing impact evaluations of STIP.

Finally, we conclude discussing other key specific issues related to the evaluation of STIP: the timing of effects, intensity of treatment, multiple treatments, impact heterogeneity, externalities, and general equilibrium effects.

Obviously a methodological review is by nature a continuous work in progress. This guideline is no exception. Therefore, any comments, suggestions and corrections are more than welcome.

Enjoy the reading!

P.S. By the way, in the same series you find other two guidelines I previously co-authored: Primer for Applying Propensity-Score Matching and Designing Impact Evaluations for Agricultural Projects. Comments are welcome on those as well!


Filed Under: Evaluation methods and techniques Tagged With: competitiveness, evaluation questions, experimental, externalities, general equilibrium effects, impact heterogeneity, innovation, intensity of treatment, multiple treatments, quasi-experimental methods, science, technology, Theory of change, timing of effects, toolkit

Alessandro Maffioli

Alessandro Maffioli es jefe de la División de Efectividad en el Desarrollo de BID Invest. Durante sus varios años de experiencia en el Grupo BID, Alessandro ha coordinado diversos proyectos de evaluación de impacto para el desarrollo agrícola, tecnología e innovación, pequeñas y medianas empresas, y acceso al crédito. Previamente, trabajó durante cinco años en el Centro de Estudios Latinoamericanos y Economías de Transición de la Universidad de Bocconi, donde se especializó en economía de la innovación y economía regional. Alessandro es doctor en economía de producción y desarrollo de la Universidad de Insubria, en Italia.

Reader Interactions

Comments

  1. Susan Kistler says

    February 23, 2012 at 7:42 am

    Alessandro, I’d love to spread the word about these (both the new one and the two previous), but I can’t get the links to work. Are there alternatives?

    Thanks!
    Susan

    Reply
    • Avatar photoFrancisco Mejía says

      February 23, 2012 at 12:03 pm

      Susan
      Thanks. It seems that the link only seemed to work inside the IDB’s networks. We will fix it ASAP.

      Francisco (I manage the blog)

      Reply
    • Pim Savetmalanond says

      February 23, 2012 at 6:02 pm

      Hi Susan,

      Can you try downloading the files again? They may take some time download but they should be working now.

      Best,

      Pim

      Reply
  2. Susan Kistler says

    February 23, 2012 at 7:42 am

    Alessandro, I’d love to spread the word about these (both the new one and the two previous), but I can’t get the links to work. Are there alternatives?

    Thanks!
    Susan

    Reply
    • Avatar photoFrancisco Mejía says

      February 23, 2012 at 12:03 pm

      Susan
      Thanks. It seems that the link only seemed to work inside the IDB’s networks. We will fix it ASAP.

      Francisco (I manage the blog)

      Reply
    • Pim Savetmalanond says

      February 23, 2012 at 6:02 pm

      Hi Susan,

      Can you try downloading the files again? They may take some time download but they should be working now.

      Best,

      Pim

      Reply
  3. klaus zinoecker says

    February 28, 2012 at 1:35 am

    link is still not workiung…

    Reply
    • Avatar photoFrancisco Mejía says

      February 29, 2012 at 9:25 pm

      Thanks. We have corrected the link. It should work now. Sorry for the inconvenience!

      Reply
  4. klaus zinoecker says

    February 28, 2012 at 1:35 am

    link is still not workiung…

    Reply
    • Avatar photoFrancisco Mejía says

      February 29, 2012 at 9:25 pm

      Thanks. We have corrected the link. It should work now. Sorry for the inconvenience!

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Follow Us

Subscribe

Search

About this blog

This blog highlights effective ideas in the fight against poverty and exclusion, and analyzes the impact of development projects in Latin America and the Caribbean.

Categories

Footer

Banco Interamericano de Desarrollo
facebook
twitter
youtube
youtube
youtube

    Blog posts written by Bank employees:

    Copyright © Inter-American Development Bank ("IDB"). This work is licensed under a Creative Commons IGO 3.0 Attribution-NonCommercial-NoDerivatives. (CC-IGO 3.0 BY-NC-ND) license and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC- IGO license. Note that link provided above includes additional terms and conditions of the license.


    For blogs written by external parties:

    For questions concerning copyright for authors that are not IADB employees please complete the contact form for this blog.

    The opinions expressed in this blog are those of the authors and do not necessarily reflect the views of the IDB, its Board of Directors, or the countries they represent.

    Attribution: in addition to giving attribution to the respective author and copyright owner, as appropriate, we would appreciate if you could include a link that remits back the IDB Blogs website.



    Privacy Policy

    Derechos de autor © 2025 · Magazine Pro en Genesis Framework · WordPress · Log in

    Banco Interamericano de Desarrollo

    Aviso Legal

    Las opiniones expresadas en estos blogs son las de los autores y no necesariamente reflejan las opiniones del Banco Interamericano de Desarrollo, sus directivas, la Asamblea de Gobernadores o sus países miembros.

    facebook
    twitter
    youtube
    This site uses cookies to optimize functionality and give you the best possible experience. If you continue to navigate this website beyond this page, cookies will be placed on your browser.
    To learn more about cookies, click here
    X
    Manage consent

    Privacy Overview

    This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
    Necessary
    Always Enabled
    Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
    Non-necessary
    Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
    SAVE & ACCEPT