Development that Works
  • About

    This blog highlights effective ideas in the fight against poverty and exclusion, and analyzes the impact of development projects in Latin America and the Caribbean.
  • Tag: impact

    Found 18 posts.

    Timely thoughts on impact evaluations

    By - 18 de March de 2014, 2:01 pm

    blog arturo tardis

    Everyone has great ideas when it comes to designing development projects or policies.

    At the drawing board, and with a bit of ideological support, an abundance of ideas are born that could easily improve countless lives. However, these ideas, while sound in theory, don´t always work.  If only there was a time machine to take us to the future to verify the effectiveness of our projects and policies before implementation.

    In the British TV show “Doctor Who”, which has been running for over 50 years (talk about an effective intervention!), the main character travels through time and space in a device called the Tardis. With a Tardis, our projects would certainly reach 100% of their development goals and we would be the super champions of development effectiveness.

    Unfortunately, economists don’t have such a device. Instead we have economic theory and empirical methods to validate some of our theories and anticipate the results of our projects. Among them are impact evaluations.

    In a nutshell, impact evaluations measure the results of an intervention for a certain group of people.  The methodologies frequently used identify changes in any relevant dimension in the population that benefits from an intervention and compares them to a group of people (preferably identical) that did not receive it. This is a way to differentiate ideas that work from the ones that don’t. Read more…

    CCTs: not the silver bullet, but with long lasting positive effects

    By - 24 de April de 2013, 5:55 am

    cct nica 1

    Conditional Cash Transfers (CCTs) were a major social policy innovation in the mid/late 1990s.

    Instead of providing generalized subsidies, using price controls, and/or directly distributing food as means to help the poor (instruments that are inefficient, distort markets and in general are regressive), governments began to transfer cash directly to the poorest families, conditioning such transfers on families sending their kids to school and taking them for routine checkups to health facilities.

    While the evaluations show that the programs had the intended effects (i.e. families increased their consumption — with no evidence of negative labor market impacts on adults–, and the use of health and education services also increased) CCTs have been under scrutiny from several perspectives.

    Some of the most common criticisms to CCTs are:

    • The impacts on “final” health and education outcomes are mixed (Fizsbein and Schady, 2009).
    • Even with CCTs there is a large enrollment gap in secondary education (which is key for the likelihood of getting a good job).
    • CCTs don’t work as well in urban areas (Bouillon and Tejerina, 2007).
    • The impact on income generating capacity of adults (by easing liquidity constraints for productive investments, which would be a positive externality as it was not part of the core objectives of CCTs), although promising at first (Gertler, Martinez and Rubio, 2006) , has been mixed (Maluccio, 2007).
    • When children from beneficiary households enter the labor market, they don’t get good jobs (Samuel Freije y Eduardo Rodriguez, 2008).
    • They don’t do enough to eliminate gender disparities.

    These criticisms address important issues, but most of these go beyond CCTs. These programs are a tool that should be part of a broader social strategy, which together with other public policies (productivity, fiscal, innovation…) would help to break the intergenerational transmission of poverty.

    What we are seeing in some cases is that the kids are not accumulating enough human capital, but this is largely because the poor quality of health and education services. Children that benefited from CCT attended school and health centers more than if their families had not received the transfer, but when they join the labor market they are still not able to find good jobs.

    This is a worrying reality, but clearly CCTs cannot be charged with finding jobs for beneficiaries in labor markets that are not generating good jobs.

    In this context, is it still a good idea to invest in CCTs? Read more…

    Do policy briefs work?

    By - 5 de October de 2012, 6:28 am

    what difference does a policy brief makeThe International Initiative for Impact Evaluation (3ie) and the Institute of Development Studies (IDS) in collaboration with the Norwegian Agency for Development Cooperation just published a rigorous assessment – based on an randomized control design- that tries answer the question:

    Do policy briefs affect readers’ beliefs?

    The answers turn out to be not that reassuring:


    Glass half empty version

    We find that the policy brief had little effect on changing the beliefs of readers who held strong prior beliefs on entering the study, but had some potential to create evidence-accurate beliefs among readers holding no prior beliefs.

    Glass half full version

    The policy brief is more effective in creating evidence-accurate beliefs among respondents with no priors, than changing beliefs of respondents who already have an opinion

    The impact of the policy brief seems to be independent of its specific form, but if it includes a research opinion and this is attributed to an authoritative source, impact is greater. Read more…

    3 good examples of the impact of impact evaluations

    By - 19 de July de 2012, 8:16 pm

    examples of impact of impact evaluationsA few weeks ago, I published a blog post on some of the unsettling implications of this paper that suggested that some interventions lose their punch when done by public agencies.

    One of the takeaways was the need to “go up the bureaucratic supply chain” as nicely put by Justin Sandefur in a tweet on the post.

    In other words, the need to jump over the “challenge of implementation” hurdle as Gabriel Demombynes described in his excellent blog.

    Just as Esther Duflo has argued that we need to understand that the environment in which people make decisions is very different for the poor than for the rich, we also need to understand that institutions (in a general sense) are endogenous in impact evaluations.

    Just as we know that most vaccines will not be as effective (and sometimes even harmful) when they are not kept frozen or refrigerated, it is important to acknowledge that many interventions could backfire when incentives differ significantly from the experimental setting.

    Understanding institutional frameworks seems critical if impact is to have any scale.

    So, how do we build that capacity to evaluate impact in challenging institutional settings?

    Let me suggest one direction. Read more…

    10 Pitfalls in Cost Benefit Analysis

    By - 7 de June de 2012, 6:04 am

    Pitfalls in Cost Benefit Analysis

    Demonstrating effectiveness is at the core of the RCT revolution (led by lab, quasi and discontinuous randomistas and followed by many of us on twitter). It is only quite recently that structural models have begun to appear that allow for reliable estimates for welfare improvements, demand curves and benefit flows.

    For instance, this recent study on de-worming in Kenya which Michael Kramer presented at the Center for Global Development estimates high Internal Rates of Return (from 22.9% for productivity gains alone to 39.3% considering total earnings). This emerging trend to embed rigorous Cost Benefit (or Cost Effectiveness) in impact evaluations is of critical importance.

    A big bang impact is probably sufficient to draw the attention of governments or NGOs. But showing a big bang for the buck is necessary to open their wallets.

    So, as this welcomed trend evolves, it is relevant to look at some of the potential pitfalls that can emerge from doing Cost Benefit analysis.

    Here is a list of ten pitfalls in no particular order Read more…