Impact Evaluation in Practice

 

Want to test if a program or policy works, but don’t have time for a Ph.D. in econometrics? Then this book is for you.

In “Impact Evaluation in Practice” we discuss the basic principles of impact evaluation and its application in the real world, all in a very accessible language.

“Impact Evaluation in Practice” is available for free on the World Bank’s website.  In it you will find links to training materials, including presentations and audio-visuals (here I am in Beijing, presenting on one of my favorite subjects: experimental evaluations).

For those of you who might want to read the book in Spanish, the launch is scheduled for June of this year in Cuernavaca, Mexico during the “Mind the Gap: From Evidence to Policy Impact” conference.

The book is divided in three parts.

The first offers an introduction to the subject, including a discussion of why impact evaluations are important and how they relate to other types of evaluation.

We discuss the process of formulating evaluation questions and hypotheses to be tested. The definition of relevant questions about the subject of interest is the key starting point for any evaluation, since all subsequent steps will be focused on answering these questions.

Did the program have an impact on final results? What were the dimensions of the impact? What components of the program work better than others? Did the program have unintended consequences?

The second section of the book concentrates on the primary impact evaluation methods, minimizing technical notation and maximizing the conceptual discussion about how different methodologies allow us to reach a common goal: estimating the causal relationship between and intervention and an outcome.

The methods include names like “Randomized Assignment of Treatment” and “Regression Discontinuity Design” and other seemingly complicated names. But when we consider the final objective of each method, we see that essentially they are all trying to reconstruct the state of the world that would have prevailed in the absence of a program (the famous “counterfactual”).

Once this objective has been accomplished, estimating the impact of a program is trivial. We simply calculate the difference between (1) the observed condition of a beneficiary when he receives the program and (2) the estimated condition of the beneficiary had he had not received the program.

The first piece of this equation is relatively easy to measure. We just need to go to the beneficiary (person, locality, school, hospital, etc) and measure the outcome indicator (related to poverty, health, education, etc). The challenge is with the estimation of (2), since we can’t observe it directly. Although all methods attempt to estimate (2), in the book we discuss some of the main advantages and disadvantages of each method.

The last section of the book touches on the more operational aspects of impact evaluations. Many of the lessons here are based on the practical experiences that my co-authors and I gained in the field. We can think of an impact evaluation as a project within a project.

To achieve a successful evaluation from start to finish you need planning, personnel, financing and a little luck. We discuss the steps required to design an evaluation within the operational rules of a program, how to assemble an evaluation team, topics related to sampling and data collection, and how to put together a budget, among other topics.

Finally, we discuss the types of publications and activities that help the evaluation have its own “impacts” in the world of public policy.