Can impact evaluations make a difference? Can they be something more than looking for the keys where the light is instead of where the keys were actually lost?
The answer is an unequivocal yes, but for reasons other than the ones usually put forth. The argument I make in this note is that impact evaluations are important not only because of their direct effect in showing which policies work and which don’t, but also because they can be instrumental in changing the political equilibrium that determines the overall quality of policies. Instituting and pursuing an evaluation culture may help build stronger government capabilities and public policies better aligned with development outcomes.
*1942 June 3, Florence Morning News, Mutt and Jeff Comic Strip, Page 7, Florence, South Carolina. (NewspaperArchive)
Impact Evaluations are Important
Most people agree that evaluating policies is necessary to find out whether they had any impact, and the direction and size of that impact. Evaluation allows for calculating the cost-benefit of scaling up or replicating the intervention elsewhere. As such, “Randomized Control Trials” (RCTs) in particular have become popular because assigning participants to the program at random increases the likelihood that any positive effects are actually due to the program. As the 2014 Development Effectiveness Report sums up, “such evaluations allow us to verify that indeed we have selected to do the right things and are doing them right. Impact evaluations show that observed positive trends or development results are because of, or attributable to, the project.” The stories about impact evaluations conducted by the IDB in 2014 offer a glimpse of its importance.
Do RCTs help answer the main development questions?
Not necessarily. Impact evaluations are only one tool for enlightening development practitioners. For example, RCTs conducted by The Abdul Latif Jameel Poverty Action Lab (J-PAL) in the area of education have helped identify many ways of increasing school enrollment and attendance, numerous strategies for improving student learning, and the wide variation in the cost-effectiveness of different programs. Still, as Ricardo Hausmann points out, while education is undoubtedly important, the impact of education on economic growth is not so clear. On average, education attainment has increased by almost 4 times over the last 50 years while income per capita has not even doubled in the same period. Moreover, countries such as China started with less education than Mexico and made fewer educational gains, but have still been able to grow much faster. Consequently, the impact of RCTs in many areas can be important but marginal to the main development questions.
Additionally, as Lant Pritchett and Philipp Krause have skillfully written, many countries have had really good outcomes without using very rigorous evidence, much less RCTs; no current developed country relied much on RCTs during its process of development. Countries that have embraced bad policies have not necessarily done so because they lacked information. There is plenty of information out there on the pernicious effect of printing excess money or running large deficits to increase rigid current expenditures that policymakers around the world ignore. Similarly, despite ample evidence that price controls tend to reduce supply, create black markets, and lead to many long-term perverse outcomes, many politicians still use them.
What characterizes the countries that do well?
One common denominator among the countries that do well is not that they conduct more RCTs but that they tend to have better public policy characteristics. As The Politics of Policies and subsequent publications have argued, to be successful, policies should be stable, so that they change in response to economic or other conditions, rather than shifts in the political winds. They should be adaptable, so they can be adjusted or replaced when needed; well-coordinated among different agencies and levels of government; well implemented and enforced, and efficient. Finally, they should be “public-regarding,” promoting the general welfare, rather than rewarding specific individuals, factions or regions. Those characteristics, the evidence has shown, are absolutely critical. A so-called “best policy” that lacks credibility and fails to encourage long-term cooperation in the political sphere may be far less effective than a less-than-ideal, but stable and well implemented one. Indeed, a credible, though imperfect policy, can have great virtue. Unfortunately, developing countries lag behind in the quality of policy features that make for long-term, effective, policymaking processes (as shown in Figure 1).
Figure 1. Policy Index across Regions of the World
Source: Franco Chuaire and Scartascini (2014).
The set of countries that do well tends to have a stable and professionalized bureaucracy, an institutionalized Congress, programmatic and institutionalized political parties, and an independent judiciary. In other words, they have specialists who can implement those policies (professionalized bureaucrats), and third party actors who can tell the government when policies are not working (working judiciary and accountability mechanisms). Moreover, governments need to care about improving public policies; that is, they internalize the long-term effect of their actions (only possible when actors have a stake in the long term and take into account the cost of current actions –institutionalized actors).
Figure 2. Government Capabilities: An International Comparison
Source: Franco Chuaire and Scartascini (2014).
So, what is the value of RCTs for developing countries?
First, in countries with low capacities, conducting pilot studies under rigorous evaluation conditions can help policymakers steer clear of underperforming policies. This is particularly important in developing countries where scarce resources have many competing uses.
Second, and this is the gist of this note, RCTs could be a way to force countries out of the bad capacities equilibrium. Too many developing countries suffer from sub-standard congresses. Their legislators lack experience, expertise, and professional staff and, in many cases, seem uncommitted to investing in the capacities of the institution to which they belong. Too many nations are served by civil services in which employment depends more on political patronage than merit or technical skills. With few exceptions, judicial independence is an oxymoron. In several countries the courts are losing even the little autonomy they once had. In many countries, political parties are a rarity and politicians change party labels every election, which reduces the probability of anyone internalizing their actions.
The international community must help the region strengthen those institutions. As has been argued elsewhere (Scartascini and Tommasi, 2014), that doesn’t mean insisting on so-called “best legislation.” Nor does it mean pouring in money for material infrastructure, like computers, that will have no effect as long as the institutions themselves are defective. Instead, it involves using expertise and financial influence to force governments to invite the opposition and as many other players as possible to the table in the interest of enduring institution building, accepting incremental progress and “safe bets,” and striving to create incentives that push political players to ignore short-term political temptations in favor of long-term consensus and cooperation. This is where RCTs and impact evaluations can make a difference. Forcing policy decisions to be made based on solid evidence derived from RCTs would generate those conditions and help to build plenty of local capacity. If policies have to be evaluated, it is less likely that they will be switched at will from one administration to the next, when they are working. Moreover, it would be easier for policymakers to resist those policies that would not work but are being forced through the system for clientelistic reasons. As the government authorities interviewed in this video attest, evaluation helps them to increase their capacity and decision making.
Hopefully, soon enough, we will be able to look for the key where the key was lost. Surely, more and better evaluation should shine a bigger and brighter light on key problems identified by other analytical and empirical research. Hopefully too, pushing for more and better evaluation in those countries with low government capabilities in which the international community has a stake may also lead to improvements in local capacity and a better long-term political equilibrium.
***
Leave a Reply