As the debate on the impact of the Millenium Villages Project (MVP) evolved, it became increasingly clear that, at least for now, there is no evidence on its effectiveness. It is also clear that part of the challenge has been that rigorous evaluation designs were absent at inception, and later efforts did not provide much evidence with their quicksand baselines or muddy controls. No silver bullet here.
These results are disappointing, as we would like real impact in a large scale.
By design, most rigorous evaluations are done at a small scale – many experimental – and their replication at a larger scale is challenging at best, particularly if scaling up requires a capillarity that very few organizations – mostly government entities – have. And when they do have that reach, many are beset by complex institutional challenges, as the Kenya teacher intervention clearly shows.
Berk Ozler recently blogged about what he called “moving from efficacy to effectiveness” and the difficulties that go with scaling: heterogeneity of impacts; moving from a particular sample to the general population; or moving from controlled conditions to general conditions.
So, can impact survive scale?
On the one hand – and most economists have at least two hands, although some are conjurers – many large scale public sector programs show effectiveness in heterogeneous and complex settings: Conditional Cash Transfers (see the PROGRESA-Oportunidades evaluations here), or early childhood interventions (see the excellent “First steps” IDB blog here), are relevant examples.
There are also private sector projects that have reached the BOP at a scale and with an impact unthinkable ten years ago. Think cell phones.
Some years ago – Robert Jensen (famously called the “Indiana Jones of economics” by Steven Levitt) demonstrated their impact on fishermen in Kerala (ungated here). And cell phones have become the platform of choice in the private provision of previously inaccessible market goods and services. Mpesa in Kenya or Dddedo in Colombia are striking examples of cell phones as vehicles for financial inclusion.
What about NGO interventions?
Most NGOs are small and lack the capacity to operate in a large scale. Nevertheless, some NGOs, both national and international, do have reach and scale, and many have adopted a rigorous evaluation agenda as part of their core business.
For instance, an ongoing evaluation (older paper here) of a program to reach the “ultra poor” by the Bangladeshi BRAC, probably one of the largest NGOs in the world, shows pregnant promise.
The ultra poor program is targeted at assisting this segment of the population (17.5%) in graduating from extreme poverty to mainstream development programs (such as microfinance) and establish sustainable livelihood improvement.
The scale up in 2011 reached half a million households at a cost of US$300 per household (Pilots in other countries such as Haiti are significantly more expensive). The program is structured to loosen restrictions such as lack of productive assets and lack of skills and information on how to use assets.
The preliminary results show that labor income increases by 50%, assets and savings also increase dramatically. Although details are not yet available, the program generates a rate of return almost double that of cash transfers alone.
In 2006, CGAP and the Ford Foundation launched an initiative to test and adapt BRAC’s approach in a diversity of countries and contexts and recently concluded that “the pilots are beginning to demonstrate that a well-sequenced, intensively monitored program combining consumption support, access to savings, livelihoods training, and an asset transfer can lead to increased consumption, asset and income diversification, and some level of empowerment.”
On the other hand, reaching scale has great complexity and context matters a lot. It poses significant challenges (organizational, political, cultural, demographic, or institutional) and showing impact at larger scale has what Jim Manzi calls “high causal density”.
The good news is that scale and impact seem to be reconcilable and the MVP results are hopefully more of an exception than the norm. Maybe its design suffers from what Manzi calls “holistic integration’.
It is an excellent documentary.
It is an excellent documentary.