Demonstrating effectiveness is at the core of the RCT revolution (led by lab, quasi and discontinuous randomistas and followed by many of us on twitter). It is only quite recently that structural models have begun to appear that allow for reliable estimates for welfare improvements, demand curves and benefit flows.
For instance, this recent study on de-worming in Kenya which Michael Kramer presented at the Center for Global Development estimates high Internal Rates of Return (from 22.9% for productivity gains alone to 39.3% considering total earnings). This emerging trend to embed rigorous Cost Benefit (or Cost Effectiveness) in impact evaluations is of critical importance.
A big bang impact is probably sufficient to draw the attention of governments or NGOs. But showing a big bang for the buck is necessary to open their wallets.
So, as this welcomed trend evolves, it is relevant to look at some of the potential pitfalls that can emerge from doing Cost Benefit analysis.
Here is a list of ten pitfalls in no particular order
- Dazed and confused by interchanging transfers and benefits (or costs)? Some payments that appear in the financial cost (benefit) streams do not represent direct claims on the country´s resources, but only reflect a resource control transfer from one sector of the society to another. Loan and interest payments, taxes, subsidies and depreciation allowances all fall into this category.
- Sweeping externalities under the rug? In many projects, externalities, particularly environmental externalities, can be a significant source of benefits or costs and should not be ignored.
- Playing billiards with second and third round effects? In some cases, a project might affect markets that are one or two steps removed. Caution should be exercised when valuing these benefits as the more you assume, the less you know for sure.
- Wishful thinking and considering employment as a benefit? Some projects have an explicit – or implicit – goal of providing employment. Employment is a cost, not a benefit as it entails the use of a scarce resource. In those cases where employment is pursued, an appropriate shadow wage rate should be used to reflect the real opportunity cost of labor.
- On the verge of a nervous breakdown projecting your numbers?. In many projects, benefits are projected into the future using ad-hoc growth rates. This bias is not a trivial matter. A recent study on cost escalation in 258 transportation projects worldwide shows that cost underestimation cannot be explained by error but by overly optimistic projections. Sensitivity analysis should not be looking for the brighter picture, but rather explaining what might happen in the rainy days.
- Confusing sensitivity and sensible analysis? Sensitivity analysis requires an understanding of cost and benefit drivers, and of the theory of change behind them. Just multiplying some cost –or benefit- by 0.7 (or whatever other random number you choose) does not provide any additional information for decision making
- Having your cake and eating it too? or the strange life of double counting benefits. Double counting of benefits can happen in one of two ways. First, when the value of an intermediate good or service is measured twice. Second, when the benefit is counted once as a stock and then again as a flow. For example, adding a higher property value that results from reduced travel times to the location along with the monetized benefit of reduced travel times. Counting both will probably make you happier but not look wiser.
- Sliding costs or looking beyond the project budget? Project costs should be total project costs to society, not only those costs that are borne by funder or funders.
- Green eggs and ham? or mixing nominal and real flows. All analysis should be made in a consistent base currency. If the discount rate is real, flows in the same currency base should be estimated and presented. If a comparison is made to similar estimations in other countries, particularly if benefits are consumption based, estimates should be PPP.
- Suffering the unbearable lightness of the counterfactual? When cost benefit analysis is used as a forward looking decision tool, it requires understanding what happens without the project. But just as not everything is guaranteed to recur infinitely, alternative(s) should draw on valid inference and external validity concerns.
NOTE: THIS POST IS BASED ON A DRAFT ECONOMIC ANALYSIS GUIDELINE FOR IDB PROJECTS THAT I AM CO-WRITING WITH PAUL WINTERS
nice, glad to see attention to basic, solid benefit cost analysis. And you are right–a big bang may be necessary, but not sufficient from a policy or stratetgic decision-making perspective. Can’t wait to see the guidelines.
Thanks Raul. The Guidelines will be out soon (though no links to Led Zepellin). Paul and I decided to make them not a “how to manual” – plenty of good ones there – but a kind of annotated topical “why is this important” guide.
nice, glad to see attention to basic, solid benefit cost analysis. And you are right–a big bang may be necessary, but not sufficient from a policy or stratetgic decision-making perspective. Can’t wait to see the guidelines.
Thanks Raul. The Guidelines will be out soon (though no links to Led Zepellin). Paul and I decided to make them not a “how to manual” – plenty of good ones there – but a kind of annotated topical “why is this important” guide.
Excellent article. In my mind, the biggest limitation of cost-benefit analysis is that while it establishes a cost per benefit of the intervention at hand, it fails to tell us if there are more cost effective ways of achieving the same benefit. Is this the best bang for my buck? That is a question that one must also keep in mind.
Thanks for your comment. It is on point. Cost Effectiveness does provide a very good sense when one wants to compare and rank costs per effect for the same effect. It will not allow you to compare with another intervention that produces some other effect. Cost Benefit will help you with that, as it monetizes it allows to compare different interventions in a common denominator. The downside is that it requires, in most occasions, additional assumptions which weaken the validity of results. But it does give you a bang for the buck glimpse
Excellent article. In my mind, the biggest limitation of cost-benefit analysis is that while it establishes a cost per benefit of the intervention at hand, it fails to tell us if there are more cost effective ways of achieving the same benefit. Is this the best bang for my buck? That is a question that one must also keep in mind.
Thanks for your comment. It is on point. Cost Effectiveness does provide a very good sense when one wants to compare and rank costs per effect for the same effect. It will not allow you to compare with another intervention that produces some other effect. Cost Benefit will help you with that, as it monetizes it allows to compare different interventions in a common denominator. The downside is that it requires, in most occasions, additional assumptions which weaken the validity of results. But it does give you a bang for the buck glimpse
Francisco:
I find your “10 pitfalls in cost benefit analysis” a very interesting summary of important issues in CBA. The revival of this methodology is welcome, especially observing some disappointing experiences in recent years with alternative approaches to analyze investment projects.
I’d like to mention a couple of aspects for your consideration. Firstly, that an essential aspect in the application of CBA is to focus it on a well defined set of alternative options to attain the development objective. Differences on CBA results among those alternatives become useful support of decision making. In the absence of alternatives CBA may indicate a “good project”, but only when comparing to alternatives it will be “the best project” based on CBA.
Secondly, I’ve found over the years a recurrent result: implementation always takes more time than expected and anticipated during analysis. Given the importance of the time factor in CBA, I would stress the recommendation of defining reasonable but realistic assumptions on this aspect.
I find that having “an annotated topical ‘why is this important’ guide” (as you define it) is very convenient and useful as a tool for IDB specialists and other practitioners. This is a long awaited document that will contribute to the quality of future IDB projects. Good for you!
Thanks Orlando. Coming from an expert like your self this is very good to hear.
Francisco:
I find your “10 pitfalls in cost benefit analysis” a very interesting summary of important issues in CBA. The revival of this methodology is welcome, especially observing some disappointing experiences in recent years with alternative approaches to analyze investment projects.
I’d like to mention a couple of aspects for your consideration. Firstly, that an essential aspect in the application of CBA is to focus it on a well defined set of alternative options to attain the development objective. Differences on CBA results among those alternatives become useful support of decision making. In the absence of alternatives CBA may indicate a “good project”, but only when comparing to alternatives it will be “the best project” based on CBA.
Secondly, I’ve found over the years a recurrent result: implementation always takes more time than expected and anticipated during analysis. Given the importance of the time factor in CBA, I would stress the recommendation of defining reasonable but realistic assumptions on this aspect.
I find that having “an annotated topical ‘why is this important’ guide” (as you define it) is very convenient and useful as a tool for IDB specialists and other practitioners. This is a long awaited document that will contribute to the quality of future IDB projects. Good for you!
Thanks Orlando. Coming from an expert like your self this is very good to hear.
Than for this Cost Benefit Analysis article