Amplifying access to knowledge: evolving our Publications Catalog using generative artificial intelligence

At Abierto al Público, for many years we’ve been convinced that natural language processing is essential for successful knowledge management in the digital age. When ChatGPT started gaining widespread attention last year, and other LLMs (Large Language Models) were made widely accessible, we had lots of ideas about how to use them.  In support of the IDB’s Institutional Strategy, which highlights the generation and sharing of knowledge as a key element to foster development impact at scale, we began working on ways to use this evolving technology to increase access to the knowledge that the IDB makes available to the world at no cost via its Publications Catalog.

A new feature of the IDB Publications Catalog

Leveraging what we have learned from previous efforts to connect people with knowledge, we worked collaboratively to create an experience that allows you to have conversations with IDB Publications, and we are proud to introduce the result.

Today, at this very moment, anyone that visits the IDB publications catalog on the internet can engage with peer-reviewed IDB Publications by asking questions in the same way that they would ask a peer. The tool will allow you to:

This incredible tool makes it easier and quicker for anyone to access and leverage the wealth of knowledge within our publications, whether you are a researcher, a policymaker, a student, or just curious.

For example, if you are interested in better understanding the region’s current economic landscape, you can visit the 2024 Macroeconomic Report in the IDB’s Publications Catalog, and ask “What are the key factors contributing to the region’s economic resilience?”.

This tool, called Seek, generates an answer that is based exclusively on the information within the publication. It also provides a linked citation in the answer so that you can navigate to the page related to the answer and read further directly in the publication.

Or, perhaps you are reading The Geography of Intergenerational Mobility in Latin America and the Caribbean (iadb.org), and are curious about the data.  In that case, you can use our new tool to ask “What data was used for the analysis in this paper?”.  In this case, the tool will generate an answer letting you know that the analysis used census data, along with additional details. 

Start your conversation

Here is a list of some of the books that you can have a conversation with right now:

• Reshaping Retirement: Navigating Latin America’s Pension Systems after COVID-19 (iadb.org)
• When Does Automation in Government Thrive or Flounder? (iadb.org)
• Dealing with Debt: Less Risk for More Growth in Latin America and the Caribbean (iadb.org)
• Political Economy of Trade Policy in Latin America (iadb.org)
• Trust: The Key to Social Cohesion and Growth in Latin America and the Caribbean (iadb.org)
• Economic Institutions for a Resilient Caribbean (iadb.org)

To see more peer-reviewed publications that have this feature visit the IDB Publications Catalog, look for Books, Working Papers, and Monographs that have “Generative AI Enabled”.  You’ll recognize them by the purple flower icon: .

Select the “View Online” option for the document language you want to interact in, and then look for the tab on the right with the purple flower. Click to open the tab and begin a conversation.

It uses retrieval augmented generation

This new tool uses an advanced natural language processing approach known as retrieval augmented generation (RAG) to leverage the capabilities of large language models, while virtually eliminating hallucinations.  This technique involves first preparing the publications text by chunking it into sentences and phrases and then converting those chunks into vector embeddings, which are stored in a vector database. 

When you visit our Publications Catalog and ask a publication a question, the tool retrieves the most relevant chunks of text from the vector database. Then it uses advanced prompt engineering techniques to generate an answer, which is then validated to ensures that it is based on that text from the IDB publication.  Finally, citation links are added, based on the text chunks that were used to generate the answer, which allow you to navigate directly to the relevant page(s) and continue reading.  This flow minimizes hallucinations, and as an added measure, when the validation cannot attribute the generated answer back to the original text, it declines to provide a response to the user.

Knowledge sharing is at our core

Every day we strive to make knowledge available and reusable.  In line with those efforts, this tool can be used by anyone.  Nothing is perfect, and the tool can make mistakes, so be sure to verify the response with information in the publications, and to also review the Terms of Use. 

Launching this tool is a demonstration of our commitment to open knowledge, an indication of where we are heading, and an achievement that makes the plethora of IDB knowledge more accessible and actionable than ever before. 

Ready to try it out? Please do! Visit publications.iadb.org now.

Have you used our publications conversation tool? Let us know your thoughts and ideas in the comments below?