One of the takeaways was the need to “go up the bureaucratic supply chain” as nicely put by Justin Sandefur in a tweet on the post.
In other words, the need to jump over the “challenge of implementation” hurdle as Gabriel Demombynes described in his excellent blog.
Just as Esther Duflo has argued that we need to understand that the environment in which people make decisions is very different for the poor than for the rich, we also need to understand that institutions (in a general sense) are endogenous in impact evaluations.
Just as we know that most vaccines will not be as effective (and sometimes even harmful) when they are not kept frozen or refrigerated, it is important to acknowledge that many interventions could backfire when incentives differ significantly from the experimental setting.
Understanding institutional frameworks seems critical if impact is to have any scale.
So, how do we build that capacity to evaluate impact in challenging institutional settings?
Let me suggest one direction.
There are many ridges to the question and it cannot be answered in one blog post. But a good initial glimpse at the answer can be had by looking into what governments around the world are doing to embed rigorous evidence into policy design and implementation.
Here are some examples.
1. In the US, many Departments in the Executive Branch – such as Health and Human Services or Justice – have been conducting RCTs as a core element in their policy formulation and implementation for many years.
In 2009, the Office of Management and Budget announced a government wide initiative to fund rigorous evaluations in federal spending. In May of this year, the White House instructed all agencies to demonstrate the use of evidence throughout their Fiscal Year 2014 budget submissions.
“The American” foreign aid agency, USAID, has also pushed hard for evidence based policy in its programs worldwide.
2. In the UK, where impact evaluations are also conducted in many individual Departments, the Cabinet Office created its Behavioural Insights Team which relies heavily on rigorous evidence. DFID, the UKs foreign aid arm, is also pushing the evidence agenda as a core element in its work as its Head of Evaluation explains in this interview.
3. In Latin America, the pioneer was Mexico. The first Conditional Cash Transfer in the world, Oportunidades – created as Progresa 25 years ago in 1997- was designed and expanded on the basis of the results of rigorous Impact and Cost Benefit Evaluations.
Since then, Progresa-Oportunidades has evolved and the model has been adapted by many other CCT programs in Latin America (and elsewhere). Evidence based policy making is permeating not only social policy as shown by the role that CONEVAL plays in Mexico. It also is becoming national policy.
Perú and Chile sought assistance from MIT´s Abdul Latif Jameel Poverty Lab to “generate innovative proposals and empirical evidence that governments can use to answer key policy questions and design and implement better public policies.” Colombia has had SINERGIA, an evaluation institutional evaluation framework for over 10 years that is at the core of national social policy.
The Mexican, Chilean and Colombian experiences are reviewed in this paper by Marie Gaarder and Bertha Briceño. If you are bored reading and are craving some video, the CONEVAL and SINERGIA experiences are discussed here.
Impact evaluations are like protons, neutrons, and electrons which acquire mass only when interacting with the Higgs field. Impact evaluations will really affect national policies when they show effects in imperfect institutions. Maybe understanding better the institutional underpinnings that make them effective is our elusive particle.