The dirty laundry of benefit estimation

The application of Cost Benefit Analysis (CBA) in public policy is widespread in developed countries. For more than 50 years major public policy decisions in the US have been subject to CBA rules. The current framework was established by Presidential Executive Order 12866 which on September 30 1993 established CBA as a pivotal element in the American regulatory framework.

While controversial in some of its applications, particularly when the valuation of benefits is not derived from market prices, Cost Benefit Analysis (or as some would say Benefit Cost Analysis) is here to stay.

 

While CBA is not as prevalent in developing countries, its use has expanded, and in Latin American countries such as Chile or Peru it is a pivotal element in policy analysis and project selection. Its rigorous application, particularly in projects that involve survey based valuation methods (contingent, travel cost, hedonic) is costly, time consuming and technically demanding.

As a short-cut, welfare estimates can be derived from previous studies in what is called “benefits transfer”, where benefit estimates from studies already completed can be systematically transferred.

Benefit Transfer estimation is in fact the default valuation method used to compute benefits of environmental impact assessments at the US Environmental Protection Agency, and is widely used by development agencies, particularly for environmental projects:

While benefit transfer should only be used as a last resort and a clear justification for using this approach over conducting original valuation studies should be provided (OMB 2003), the reality is that benefit transfer is one of the most common approaches for completing a Benefit Cost Analysis at EPA.

So, are benefit transfers a clean technology or are they the dirty laundry of environmental economics?

The critical element in benefit transfer is the definition of the transfer values of additional benefit units. There are several approaches.

First, Willingness to Pay point estimates can be transferred using unit value transfers. This could be called the rule of thumb benefit transfer method: the amount that a household in Paris (Texas) is willing to pay could be used to estimate how much a similar household is willing to pay in Paris (France). This method typically relies on a single estimate [Always remember that economics is the art of making reasonable assumptions].

Second, instead of the values themselves, one can transfer the estimated function from Calcutta (India) to Cucuta (Colombia). This is called a function transfer. With this methodology, one can apply the Willingness-to-Pay function from the case study to the target population in the policy case. Third, a meta-analysis can use multiple valuation studies to estimate a new transfer function.

In any case, there is room for Benefit Transfer error when defined (ungated here) as :

”the difference between a benefit measure estimated using original data (i.e., the policy case) and a surrogate for that benefit measure based on preexisting estimates (i.e., the study cases)”.

These errors can come from differences in observables from both populations, from the methodological choice in valuation methodology and from the transfer procedures themselves.

A recent study (previous ungated version here) on Benefit Transfer reviewed 40 Benefit Transfer studies (31 were eventually used). These studies are mostly from the US and Europe and cover topics such as access to recreation sites, land preservation, changes in water quality and quantity and exposures to human health risks.

In total, the authors reported more than a thousand benefit transfer errors (1,047 to be exact), from which they derived five main conclusions:

1. The median absolute error in benefit transfers is 39%; and in more than a third – after eliminating outliers – the error is more than 100%

2. Function transfers outperform value transfers;

3. Geographic site similarity is important for value transfers;

4. Contingent valuation generates lower transfer errors than other valuation methods; and

5. Combining data from multiples studies tends to reduce transfer errors

In addition to benefit transfer error (which, by the way, tends to overestimate benefits more than underestimate them), bear in mind that any estimation of WTP by indirect methods is in itself prone to plenty of error and positive bias (more on this in a future blog entry).

It makes you wonder of external validity assumptions in impact evaluations.