How to Evaluate the Impact of Investment in Knowledge with a Dual Approach

At the Inter-American Development Bank (IDB), investing in the continuous strengthening of the capabilities of our teams and the executing agencies in member countries is a cornerstone of our mission. Training programs, such as workshops on the Environmental and Social Policy Framework (ESPF), are essential to ensure quality and minimize risks in Bank-financed projects.
Given the considerable volume of resources and time invested, a critical question arises for efficient management and strategic decision-making: How do we measure the actual economic value and tangible impact of these investments in knowledge and training? Furthermore, how do we rigorously evaluate the economic return and attributable impact of these investments on development effectiveness? The purpose of this blog, then, is to delve into how well-established methodologies such as cost-benefit analysis (CBA) and Counterfactual Impact Evaluation could help us address these questions.

CBA: A First Approach to the Economic Value of ESPF Training

CBA is a financial methodology used to assess the economic viability of various initiatives by comparing costs with expected benefits over a defined time horizon. Although it may seem complex, this analysis, when applied to a training program like ESPF, can be broken down into the following core components:

1. Quantification of program costs: This phase requires gathering all the accurate information, which can be laborious. It involves identifying and adding up all the financial and economic resources invested in the training program over a specific period:
2. Direct Costs: These include the fees of agencies, instructors and/or external consultants, the costs for development, design, and production of training materials; travel and logistics expenses for in-person workshops; and the costs of virtual training platforms.
3. Indirect Costs: These include the cost of time that bank professionals and borrowers dedicate to training instead of operational responsibilities. This opportunity cost must be estimated and considered.

The formula to obtain the total cost of the program would be:

According to cost theory, indirect cost is one that cannot be exclusively attributed to the product or service. It includes utilities and staff salaries, whether specialists or administrative personnel not directly working on the program but supporting it, only if these costs have not been otherwise allocated.

Beyond the Benefits: Avoided Costs

• Benefit Estimation: In a training program like MPAS, benefits rarely appear as direct monetary income. Instead, we speak of avoided costs in the preparation and execution of IDB-financed projects, thanks to the more efficient and effective application of environmental and social standards. Some examples of these costs include:

• Reduction of project schedule delays linked to shortcomings in environmental and social management.
• Decrease in costly redesigns during advanced project phases, due to better risk and impact identification in early stages.
• Minimization of financial, temporal, and reputational costs associated with managing social conflicts or complaints from affected parties, because of better consultation processes and improved community engagement.
• Prevention of fines, sanctions, or disbursement suspensions due to non-compliance with Bank policies.
• Optimization of time for both Bank staff and the executing agency in supervision and problem-solving tasks.

How much is due to training?

The fundamental challenge when conducting an independent CBA is to estimate how much of these savings are directly attributable to training. Usually, this estimation is based on:

• Expert judgment: Consultations with panels of specialists, team leaders, or evaluators/actuaries to obtain reasonable estimates on the expected reduction of these problems because of training. For example, “a reduction of X% to Y% in delays is estimated due to improved understanding and application of the ESMP.”
• Scenario analysis: Multiple scenarios development (the classic ones are: low, medium, high) to reflect the inherent uncertainty in the effectiveness of the training and its translation into avoided costs.

Calculating Economic Profitability Indicators

Once the projected annual flows of costs C_t and the estimated benefits B_t are available for each effectiveness scenario, the standard economic evaluation indicators are calculated:

• Net Present Value (NPV): Compares the present value of expected future benefits with the costs incurred, using a discount rate (r) that reflects the opportunity cost of capital for the IDB. A positive NPV suggests an economically justifiable investment under the adopted assumptions.

• Internal Rate of Return (IRR): This is the discount rate at which the NPV of the program would be zero. In contexts with non-conventional cash flows (i.e., with multiple sign changes), multiple IRRs may exist, so interpretation must be done with caution. If the IRR is unique and exceeds the IDB’s reference rate, the program is considered an economically attractive investment.
• Benefit-Cost Ratio (BCR): This metric measures the monetary return in benefits for each monetary unit invested, both expressed in present value terms. A BCR greater than 1 is considered favorable.

Limits of CBA: The Problem of Attribution

CBA provides a structured and quantitative framework. This tool compels us to think systematically about the resources invested and the expected outcomes. In fact, CBA is a standard practice in feasibility assessments for development projects. A recent example is the World Bank’s 2023 publication, “Assessing the Benefits and Costs of Nature-Based Solutions for Climate Resilience: A Guideline for Project Developers”. This guide not only addresses methodological approaches to CBA, but also presents case studies to guide stakeholders involved in development projects (particularly Case 8: Coastal Resilience in Emergency Recovery: Beira, Mozambique).

However, when benefits depend critically on the effectiveness of training to change behaviors and prevent problems, the strength of the CBA’s conclusions is compromised. That strength is inherently tied to the quality and realism of the initial assumptions (and/or external judgments) regarding such effectiveness. If effectiveness is assumed rather than measured, we face a level of uncertainty that may undermine the entire evaluation.

This is where the analysis reaches a crossroads: Are we building on solid ground or on sand? How can we know whether the estimated benefits can truly be attributed to the intervention?Bottom of Form

What if we gave Cost-Benefit Analysis superpowers? Strategic integration of Counterfactual Impact Evaluation (CIE)

What would happen if, instead of relying solely on assumptions and estimates, we used rigorous empirical evidence? This evidence could be generated through tools and methods aligned with the IDB’s Impact+ strategy. That’s where a complementary approach comes into play: CIE, which acts as a catalyst and can enhance the analytical power of traditional CBA.

CIE aims to measure the net and attributable effect of an intervention (in this case, the MPAS training program) on observable outcomes in project execution, using econometric methodologies such as Difference-in-Differences (DiD), Regression Discontinuity, or Propensity Score Matching (PSM), among others. These tools allow us to isolate the effect of training from external factors that could also influence results.

Example: Difference-in-Differences in Action

For example, the DiD method compares the change in an outcome over time (before and after the training) between a group that received the training (treatment group) and a group that did not (control group). Conceptually, if we were to estimate this effect using a regression model, the equation could be simplified as follows:

Where:

Outcome_it​: Observed outcome for unit i at time t.

Post_it​: This variable equals 1 if it is the post-intervention period, and 0 if it is the pre-intervention period.

Treatment Group_i​: This variable equals 1 if the unit belongs to the treatment group and 0 if it belongs to the control group.

Treatment Group_i * Post_t​: Interaction between the treatment group and post-intervention period variables, capturing the differential effect of the intervention over time.

Beta₃: Coefficient representing the causal effect of the treatment, i.e., the estimated impact of the intervention.

Beta₀, Beta₁, Beta₃: These coefficients capture fixed effects and general trends.

Epsilon_it: Random error term.

This model would let us isolate the effect of the intervention by controlling for time-invariant differences between groups and time trends that affect both groups equally. The logic of an CIE is similar to a clinical trial in medicine, where a treatment group is compared to a control group to determine the causal effects of a drug. For example, one possible conclusion could be:

“Participation in the MPAS training workshops resulted in a statistically significant average reduction of X days in project delays directly attributable to environmental and social management.”

This quantitative result (X days) represents a measurable change in an observable metric k, causally attributable to the training program: Delta Impact_k​. This information is analytically valuable, as it allows direct empirical evidence to be translated into evidence-based monetary estimates within a CBA.

Analytical Synergy: When CBA Meets Causal Impact Evidence

Combining the findings of a CIE with the structure of a CBA leads to a more robust, accurate, and defensible economic evaluation:

1.  Evidence-Based Benefit Estimation: The Delta Impact_k becomes a key input for quantifying total benefits, paired with the unit value as follows:

Where:

Delta Impact_k represents the observed change causally attributable to the program (reduction in delay days, number of incidents, number of complaints, etc.).

Unit Value_k represents the cost associated with one unit of that change (average cost per day of delay or per incident avoided).

• Empirically Grounded Economic Indicators: Final indicators (NPV, IRR, and BCR) of the training program are now calculated using benefits anchored in empirical measurements.
•  Sensitivity Analysis with Statistical Confidence Intervals: Confidence intervals accompanying impact estimates allow for more objective scenario analyses. For instance, if it is estimated that “delays were reduced by between 3.2 and 6.4 days with 95% confidence,” this range defines the lower and upper bounds (through statistical formulas) and can support improved scenario planning.

Final Strategic Considerations

Adopting this combined approach to evaluating training programs such as MPAS yields significant strategic benefits for the institution:

• Optimization of financial and human resource allocation
• Continuous improvement and evidence-based program design
• Strengthened accountability and transparency (by tangibly demonstrating the value generated by the Bank’s investments in knowledge and capacity building)
• Maximized contribution to development effectiveness.

This discussion around combining impact evaluation and cost-benefit approaches is not new. In fact, it was raised over a decade ago in institutional spaces such as the Development Effectiveness blog. Today, that vision is being expanded with more ambitious tools like HUELLA, a methodology developed by the IDB that uses satellite imagery and quasi-experimental designs to measure project impacts. makes it possible to assess, at scale, the cumulative effects of multiple interventions including dimensions such as economic growth or environmental sustainability through non-traditional indicators.

Innovations That Enhance Impact Measurement

In the area of capacity development, complementary innovations have also emerged, such as An Integrated Approach to Impact Evaluation and Recognition of Learning (MEiRA). Recently implemented in flagship courses such as the one on Environmental and Social Performance Standard 1, this framework enables the evaluation not only of knowledge acquisition, but also of its practical application and real-world impact. It does so through an approach based on five pillars: from learner engagement to knowledge transfer.

These methodological advances allow the Bank not only to estimate the value of its interventions but also to demonstrate it with rigor. In doing so, they strengthen evidence-based decision-making, optimize the use of scarce resources, and, most importantly, help highlight and amplify the positive impact of our work across Latin America and the Caribbean.