Demonstrating effectiveness is at the core of the RCT revolution (led by lab, quasi and discontinuous randomistas and followed by many of us on twitter). It is only quite recently that structural models have begun to appear that allow for reliable estimates for welfare improvements, demand curves and benefit flows.
For instance, this recent study on de-worming in Kenya which Michael Kramer presented at the Center for Global Development estimates high Internal Rates of Return (from 22.9% for productivity gains alone to 39.3% considering total earnings). This emerging trend to embed rigorous Cost Benefit (or Cost Effectiveness) in impact evaluations is of critical importance.
A big bang impact is probably sufficient to draw the attention of governments or NGOs. But showing a big bang for the buck is necessary to open their wallets.
So, as this welcomed trend evolves, it is relevant to look at some of the potential pitfalls that can emerge from doing Cost Benefit analysis.
Here is a list of ten pitfalls in no particular order
- Dazed and confused by interchanging transfers and benefits (or costs)? Some payments that appear in the financial cost (benefit) streams do not represent direct claims on the country´s resources, but only reflect a resource control transfer from one sector of the society to another. Loan and interest payments, taxes, subsidies and depreciation allowances all fall into this category.
- Sweeping externalities under the rug? In many projects, externalities, particularly environmental externalities, can be a significant source of benefits or costs and should not be ignored.
- Playing billiards with second and third round effects? In some cases, a project might affect markets that are one or two steps removed. Caution should be exercised when valuing these benefits as the more you assume, the less you know for sure.
- Wishful thinking and considering employment as a benefit? Some projects have an explicit – or implicit – goal of providing employment. Employment is a cost, not a benefit as it entails the use of a scarce resource. In those cases where employment is pursued, an appropriate shadow wage rate should be used to reflect the real opportunity cost of labor.
- On the verge of a nervous breakdown projecting your numbers?. In many projects, benefits are projected into the future using ad-hoc growth rates. This bias is not a trivial matter. A recent study on cost escalation in 258 transportation projects worldwide shows that cost underestimation cannot be explained by error but by overly optimistic projections. Sensitivity analysis should not be looking for the brighter picture, but rather explaining what might happen in the rainy days.
- Confusing sensitivity and sensible analysis? Sensitivity analysis requires an understanding of cost and benefit drivers, and of the theory of change behind them. Just multiplying some cost –or benefit- by 0.7 (or whatever other random number you choose) does not provide any additional information for decision making
- Having your cake and eating it too? or the strange life of double counting benefits. Double counting of benefits can happen in one of two ways. First, when the value of an intermediate good or service is measured twice. Second, when the benefit is counted once as a stock and then again as a flow. For example, adding a higher property value that results from reduced travel times to the location along with the monetized benefit of reduced travel times. Counting both will probably make you happier but not look wiser.
- Sliding costs or looking beyond the project budget? Project costs should be total project costs to society, not only those costs that are borne by funder or funders.
- Green eggs and ham? or mixing nominal and real flows. All analysis should be made in a consistent base currency. If the discount rate is real, flows in the same currency base should be estimated and presented. If a comparison is made to similar estimations in other countries, particularly if benefits are consumption based, estimates should be PPP.
- Suffering the unbearable lightness of the counterfactual? When cost benefit analysis is used as a forward looking decision tool, it requires understanding what happens without the project. But just as not everything is guaranteed to recur infinitely, alternative(s) should draw on valid inference and external validity concerns.
NOTE: THIS POST IS BASED ON A DRAFT ECONOMIC ANALYSIS GUIDELINE FOR IDB PROJECTS THAT I AM CO-WRITING WITH PAUL WINTERS