In a previous previous blog, I explored the positive impact of impact evaluations and impact evaluation institutional frameworks have had on policy formulation.
And in a previous blog, I discussed two examples of areas where one would hope that investments and policy reflect the evidence, but do not: migration and investing in young males in urban areas.
Two examples in which, although there is ample evidence as to their development impact, public policy does not reflect it.
As I had mentioned earlier, many people buy organic food on the belief (faith, bias, hunch, impulse) that it is healthier, despite the fact that there is little or no evidence to that effect. We believe that the project/policy has an effect, the reality is that it does not, and with flimsy or little evidence, institutions chug along. Like the Energizer bunny.
So let me give you two examples, of areas in which although there is good evidence on what does not work, they still are in vogue. We could call them false positives: computers in schools and producer certification.
In previous entries we have argued in this blog that rigorous evaluations of ICT programs like One laptop Per Child yield no significant impacts on educational outcomes such as test scores.
Although in developed countries the evidence suggests a positive relationship between computer use and test scores, in developing countries the evidence indicates that the introduction of technology per se does not change the teaching and learning process, and has no significant impact on educational outcomes. Positive outcomes are possible when technology is linked to changes in pedagogy.
The Colombian program Computers to Educate, created in 2002, refurbishes computers donated by the private sector and donates them to public schools. In 2011, it donated over 80,000 computers in 8,600 schools in the country.
A recent rigorous evaluation (HT @felbarrera) has shown that although the program increases the number of computers in schools, it has little impact on scores in Spanish or math tests, mostly because – at the time of the evaluation – computers were only used to teach students computer skills.
These results provide “a sobering example of the potential limits of ICT interventions aimed at improving the methods that teachers use in the classrooms”. An implementation failure, as called by the authors.
Producer certification is increasingly popular, particularly when it shows the (wealthier) consumer that the (poorer) producer adheres to defined environmental standards and that the producer´s social welfare is improved. In theory, these programs create incentives for producers to improve their environmental, social and economic performance.
A recent comprehensive review of the impact of certification in areas where it is prevalent (bananas, coffee, fish products, forest products and tourism operations) shows two things.
First, that the quality of the studies is problematic, as only 11 studies – out of 213- used methods that generated credible results. Among these studies, 9 examined the economic effects and 2 the environmental effects of certification.
And second, that the evidence itself is also problematic. The 11 studies produced very weak evidence for the hypothesis that certification has positive effects at the producer level. Seven of them did not show any effects at all and four provided evidence, although this evidence is rated as problematic and inconsistent.
It must be noted that there is plenty of evidence that sustainable agricultural practices do have an impact on environmental and social indicators. It’s the certification itself that is questioned by this study.
So next time you drink that certified java in your local coffee shop, it might not taste as good as you thought it did.
Leave a Reply