At the closing session of the Mind the Gap: From Evidence to Policy Impact conference that took place on June 17th, the Chilean Planning Minister Felipe Kast made some interesting points about how to persuade politicians to embrace evaluations.
His perspective is interesting as until last year he was an academic engaged in impact evaluations, and today he is one of the policymakers that evaluators want to influence.
Minister Kast suggested that we need to understand at least three things from politicians:
- They like to tell good stories, but they
- Lack credibility and they also
- Hate Criticism
According to Kast these three features can be used to leverage an impact evaluation agenda.
Impact evaluations may provide politicians with good stories, and if these stories are based on solid research they will be credible.
On the other side, the politician’s interest in avoiding criticism provides evaluators with a tool to catch their attention: publicly criticize lack of interest on evaluations, question why large amounts of money are being spent without measuring their impact.
The essence of this story is often illustrated with the Mexican conditional cash-transfer program Oportunidades.
This program had a rigorous experimental evaluation that showed positive impacts on consumption, school attendance and visits to health centers. Hence, the program was not cancelled when the government changed, and now benefits 5 million families.
Furthermore, the evaluations have allowed making gradual adjustments to the program in order to increase its impact. The evaluations of Oportunidades have allowed many politicians and evaluators to tell good and credible stories.
However, what happens when evaluation findings are not entirely good news?
Let us consider two stylized facts about evaluations and development interventions:
- Some programs do not work
- Most programs that work have meaningful but small impacts
If these facts were not true, development economics would be part of economic history and we would not be talking about meeting the Millennium Development Goals in 2015. The first claim is supported by Rossi’s Iron Law of Evaluation: The expected value of any net impact assessment of any large scale social program is zero. The second point is illustrated with many examples in Poor Economics, that shows how successful programs such as microcredit that have meaningful impacts (a 10% increase in consumption in one example), but that clearly will not make poverty disappear (the point being here that there are no magic bullets).
This means that evaluators need to be cautious about the stories that they provide to policymakers, in order not to create false expectations about the potential impact of policies.
Oportunidades provides an example on the need to be cautious and realistic. Ten-year evaluations of the benefits of the program show that, beyond the increase in consumption that follows the cash transfers and has direct impacts on current poverty (not a minor achievement as it is an efficient way of transferring resources without the deadweight loss and distortions caused by subsidies), the program had an impact of increasing education and, for the most disadvantaged kids, it increased height by one centimeter.
These are important results, and have resulted in children to have higher human capital than what they would have had in the absence of the program.
Oportunidades will not solve poverty, and it was not intended to do so. It successfully contributes to this goal, and it has also generated evidence on the need to improve the quality of health and, in particular, education.
These are important stories that do not necessarily fit the scheme of providing good credible stories that will prevent criticisms of the type that politicians like (they would like to tell fairy-tale like stories about magic bullets).
So, the challenge remains on how to make good evaluations influence policy, even when the results are not always positive.
Leave a Reply