By Gastón Gertner and Lucas Figal Garone*
In one of the case study panels in our Surveys and Impact Evaluation Workshop in Santiago this week (#Taller_EEiPP), we discussed a topic that is clearly not trivial for Impact Evaluations. How can we implement a random assignment of beneficiaries when the possibility of doing so at a desk or an office with computer-programmed instructions is limited?
Impact evaluation courses place a lot of emphasis on the importance of randomly assigned treatment and control groups in order to be able to compare.
Nevertheless, once the class is over and participants return to their projects, doubts arise as to how to implement this design in real life.
In our workshop we shared some successful experimental designs where individual or group assignment to treatment or control in interventions in Bolivia and Dominican Republic was done via public raffles.
In Bolivia, the Water and Sanitation Program for Small Localities and Rural Communities includes community level interventions.
The evaluation had an experimental design. In the specific context of Bolivia, with empowered rural and indigenous communities, local authorities required that random assignment had the approval and consensus of the beneficiaries.
Within that context and in eligible municipalities, specific community and project participation and sequencing was decided in public raffles.
In the Dominican Republic, the evaluation of the Programa de Apoyo a la Transición Competitiva Agroalimentaria Nacional de Extensión Agrícola was also experimental.
The national government approved the experimental strategy as this was a mechanism to handle excess demand. Budget restrictions limited the number of beneficiaries to 8,000, out of 22,000 eligible producers. In addition, beneficiary selection had several restrictions.
The process had to be adapted to the existence of several technologies, public sector institutional limitations in their delivery, regional balance and others.
In both projects, random assignment was successful. In Bolivia, community raffles were done in buckets that contained the names of all of the communities. Each representative from eligible communities took turns extracting names, which also allowed to rank and prioritize local works in the communities.
In PATCA II, the lottery was implemented with raffle drums in two stages. In the first stage, eligible and non eligible communities were numbered and selected. In the second stage, individual producers (beneficiaries and non-beneficiaries) were assigned by region and technology using the last digit in the producer´s ID card.
A successful field randomized assignment requires:
- The preparation of a protocol, an action plan and transparent rules for each draw
- The promotion and encouragement of local and beneficiary participation in the events à This guarantees transparency
- The development of a communications strategy that includes the dissemination of clear eligibility criteria and messages that underline the fairness and equitable nature of a random draw in a raffle or lottery mechanism
- Working closely with executing agencies and authorities at all levels of government
- Having a field team that knows the area well
- The legitimization of the random results in order to establish consensus among the different stakeholders à for instance, by having a public notary in the process
- Safeguarding participation in the program according to the draw results à This requires, in turn, being efficient in the effective use of funds
- Managing crisis, political demands and social pressures that can arise and contaminate the impact evaluation
The participants in the workshop consulted on the potential use of random draws in their individual projects. Each place is different and this has to be considered when creating consensus on random assignment of treatment and controls.
The discussion of lessons learned on the field is always an original way to feed the discussion in order to construct creative solutions to topics that seem trivial but are not.
Leave a Reply