A few weeks ago, Martin Ravallion wrote a great blog on “shoe-string” evaluations, in which he suggested that there are ways of bypassing two of the most costly items in rigorous Impact Evaluations: contemporaneous credible baseline data; and second, “objectively assessed outcome indicators”, which are based on surveys.
Ravallion then proposed a “shoestring” solution that replaces costly quantitative contemporaneous baseline data, with retrospective qualitative data gathered by asking post intervention treated and untreated groups on how much their welfare has improved since the intervention began.
To test the method, Ravallion then applied it to a full blown IE in China and found it unreliable. Being this a test where n=1, he then calls for additional such tests, given that their marginal costs should be very low and have great potential.
So it seems the jury is still out on reaching credibility inexpensively.
Or is it? Maybe not, if we take look at the potential of administrative data.
That is, instead of using customized survey information specifically designed to capture expected outcomes via predefined outcomes indicators, it is often possible to reduce costs by using existing administrative data that has already been collected for other purposes such as health records, criminal arrests and test scores, for example.
To illustrate the feasibility – and flexibility – of this approach, the paper summarizes five RCTs, in areas as diverse as criminal justice (here and here), child welfare and substance abuse, community wide interventions to improve parenting skills, and teacher incentive systems.
The paper also highlights that certain conditions must be met in order for this type of low cost RCT to be feasible. The first one is obvious, access to quality data on outcomes has to be cheap. Second, a sizeable number of individuals are available without special recruitment efforts, and third, key policy officials have to be on board to assure access over time.
In addition to the examples listed above, there are many good applications of RCTs using administrative data. Alessandro Maffioli reported here on the potential use of administrative data and that there is a real “island of misfit toys” for evaluators out there in the evaluation of innovation programs. An island that can even be used to document the impact of land distribution on political power in 16th century Buenos Aires.
Note: the Coalition for Evidence based Policy has an outstanding publication list which includes not only technical papers and reviews, but comprehensive Guides and Checklists (great for those of us who love the Cool-Aid of the Checklist Manifesto) that are comprehensive, simple and extremely useful.