Development effectiveness is at the core of many programs and projects. But as Alexandre Meira da Rosa, manager of the Office of Strategic Planning and Development Effectiveness at the IDB, pointed out, achieving intended outcomes does not always occur in a straight line.
Development programs have traditionally been conceptualized as sets of products that are delivered to positively affect the living conditions of a population. In the education sector, for example, it was traditionally assumed that implementing a teacher training program, providing new textbooks, or giving laptops to students would be enough to improve student learning. However, decades of research and practical experience by development actors have shown that this is not always the case. A frequently cited example is a program that provided textbooks to primary schools in rural areas of Kenya in the late 1990s. The children in those schools had previously lacked access to textbooks. An experimental evaluation, however, found that providing them failed to raise average student achievement during the first three years of the program’s implementation. Thanks to this evaluation, the funders, the government and the executing agencies discovered that many students, especially the youngest and poorest, could not read or use the textbooks effectively because they were written in English. Although English was the official medium of instruction in Kenyan schools, it was not the first or even second language for most students, but their third one. The program did not achieve the expected outcomes. But its timely, results-oriented evaluation helped uncover a deeper problem in the education system that textbook distribution could not solve. [i]
In recent decades, the IDB and other Multilateral Development Banks (MDBs) have promoted the design and implementation of results-oriented development projects to minimize these types of failures. Such projects are rigorously evaluated, based on the achievement of their outcomes. For example, projects funded by the IDB must prove that their design has a vertical logic, or theory of change—a set of assumptions about the cause-effect relationships between inputs, activities, and products or outputs expected to lead to the intended results or outcomes. The cause-effect relationships must be based on qualitative and quantitative evidence, and evaluations must show that the projects are effective.[ii]
Over the past decade, MDBs have also developed and implemented results-based financing instruments, to strengthen incentives and accountability, including results-based loans.[iii] In contrast to a regular loan that is disbursed with the implementation of project inputs and the delivery of products, a Loan Based on Results (LBR) is disbursed on the basis of project results, irrespective of the inputs needed.[iv] A credible verification mechanism, often independent, verifies these predefined results. In this way, the LBR modality encourages development actors to monitor and focus on impactful achievements, rather than the mere completion of activities and delivery of products. Moreover, it allows for timely adjustments during the implementation of the project to accomplish the intended goals.
The IDB introduced its current LBR instrument in 2016, following a comprehensive analysis of past experiences and the incorporation of insights gained from other organizations. Since that time, it has approved 29 LBR operations totaling $4.7 billion in sectors such as health, small and midsize enterprises (SMEs), urban development and housing, social inclusion, and education, among others. These operations represent a small fraction of IDB lending but are increasingly in demand by Latin American and Caribbean governments interested in introducing new incentives and accountability measures in the public sector.[v]
Early evidence of the performance of the IDB’s LBR operations compared to other instruments is promising. Three of such operations have already been independently evaluated in Project Completion Reports (PCR), and all of them have achieved an overall rating of successful.[vi] Moreover, their PCR effectiveness ratings, which assess the extent to which they achieved their objectives as measured by previously established outcome indicators, have been satisfactory or excellent, outperforming average ratings for IDB operations. The effectiveness of these projects can be explained by their sound vertical logic in design, the evaluation of at least 90 percent of their outcome indicators, with average achievement rates of at least 80 percent per development objective, timely execution, and strong evidence of attribution of results to project implementation.
The recent adoption of results-based lending instruments by MDBs has already yielded some important lessons. These are discussed in an IDB technical note by Puerta et al. (2023). First, the definition of result indicators varies across MDBs, with most allowing disbursements against outputs (such as the construction of a school) as opposed to outcomes (such as reduced dropout rates or increased learning). The IDB instrument, by contrast, allows disbursements only against outcome indicators, with the only exception being complementary products to strengthen institutional capacity. These differences can make comparing the effectiveness of instruments across MDBs difficult and highlight the importance of a multi-sectoral agreement on definitions, as well as a renewed focus on outcomes to promote project effectiveness and impact.
Another common difficulty faced by MDBs is related to the costing and auditing of project expenditures. This is mainly due to the low level of costing detail required by LBR regulations compared to other instruments, which can compromise the project’s efficiency measurement. The IDB has increased the detail required on costs for LBRs over the years. Although outputs are not tracked during implementation, LBR proposals must present cost estimates for outputs, activities and actions identified in the chain leading to the expected results. Tracking of expenditures, although not required for disbursement, is done annually, and this information can be used to adjust the scope of a project during its execution.
Finally, LBRs require high institutional capacity and robust monitoring and evaluation mechanisms. This could exclude the most vulnerable countries and underscores the value of addressing such issues from a project’s design while providing governments with technical support. An LBR proposal at the IDB must include an institutional capacity assessment to generate inputs for the governance structure of the project. It must identify the capacity gaps that have the greatest impact on the project and plan cost-effective responses.
While development effectiveness demands tangible results that improve the lives of project beneficiaries, achieving them has been difficult. As suggested by early evidence at the IDB, the LBR instrument promises a straightforward and effective way to make the development road easier by shifting incentives from the delivery of products to the achievement of results.
[i] The impact evaluation also found that the textbooks appeared to have raised only the scores of the strongest students, with higher pretest scores. This led to the conclusion that there was a large mismatch between the official curriculum for which the textbooks were designed and the needs of many Kenyan students.
[ii] In addition, when a project’s proposed solution in terms of design is an innovation or there is insufficient existing evidence to support the expected results, a robust impact evaluation must be proposed to accompany the implementation.
[iii] Other results-based financing instruments include results-based budget support and carbon finance. For an in-depth explanation see: Kükenshöner Christina, and and Sarah Thoma. 2023. “Results-Based Financing Peer Analysis.” KfW Development Bank. Frankfurt am Main, Germany.
[iv] For an in-depth explanation see: Puerta, Juan Manuel, Germán Ferreyra, Alejandro Pablo Taddia, and Francesca Castellani. 2023. “Development Lending for a New Reality: The Evolution of Financing Instruments across Multilateral Development Banks.” Washington, D. C. https://doi.org/10.18235/0004762.
[v] The IDB has had various other successful experiences with results-based financing initiatives, most notably Salud Mesoamericaand the Social Impact Bonds (SIB) Facility established by IDB Lab.
[vi] The PCR overall performance rating at closure is a rule-bound weighted average of scores for four criteria: relevance, effectiveness, efficiency, and sustainability. Although the PCR starts as a self-evaluation of project performance by the IDB, it is then independently validated by the Office of Evaluation and Oversight (OVE).
Leave a Reply