# Evaluation of search-enabled pretrained Large Language Models on retrieval tasks for the PubChem database

## TL;DR

This paper explores how ChatGPT-4o can be used to retrieve data from the PubChem database, showing that it can effectively generate programmatic access.

## Contribution

The study introduces a methodology for using a search-enabled LLM to access PubChem, demonstrating improved retrieval accuracy.

## Key findings

- Instructing ChatGPT-4o to generate programmatic access yields correct answers more frequently.
- A methodology was developed to adapt existing PubChem protocols into LLM prompts.
- Iterative prompt refinement enhances the LLM's ability to retrieve accurate data.

## Abstract

Databases are indispensable in biological and biomedical research, hosting vast amounts of structured and unstructured data, facilitating the organization, retrieval, and analysis of complex data. Database access, however, remains a manual, tedious, and sometimes overwhelming, task. The availability of Large Language Models (LLMs) has the potential to play a transformative role in accessing databases.

We investigate in this study the current state of using a pretrained, search-enabled LLMs (ChatGPT-4o), for data retrieval from PubChem, a flagship database that plays a critical role in biological and biomedical research. We evaluate eight PubChem access protocols that were previously documented. We develop a methodology for adopting the protocols into an LLM-prompt, where we supplement the prompt with additional context through iterative prompt refinement as needed. To further evaluate the LLM capabilities, we instruct the LLM to perform the retrieval. We quantitatively and qualitatively show that instructing ChatGPT-4o to generate programmatic access is more likely to yield the correct answers. We provide insightful future directions in developing LLMs for database access.

All text used to prompt ChatGPT-4o is provided in the manuscript.