Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs

TL;DR

This paper presents a comprehensive high-level methodology for constructing AI agents that enhance Large Language Models with API integration, enabling more autonomous and context-aware real-world applications.

Contribution

It introduces a 7-step framework for empowering LLMs with API capabilities, including task decomposition, data generation, and API selection heuristics, along with reviewing existing tools and proposing an on-device architecture.

Findings

Effective API integration improves LLM utility in real-world tasks.

The proposed methodology enables more autonomous and robust AI agents.

On-device architectures with small models can be effective for API-based tasks.

Abstract

Large Language Models (LLMs) have revolutionized various aspects of engineering and science. Their utility is often bottlenecked by the lack of interaction with the external digital environment. To overcome this limitation and achieve integration of LLMs and Artificial Intelligence (AI) into real-world applications, customized AI agents are being constructed. Based on the technological trends and techniques, we extract a high-level approach for constructing these AI agents, focusing on their underlying architecture. This thesis serves as a comprehensive guide that elucidates a multi-faceted approach for empowering LLMs with the capability to leverage Application Programming Interfaces (APIs). We present a 7-step methodology that begins with the selection of suitable LLMs and the task decomposition that is necessary for complex problem-solving. This methodology includes techniques for generating training data for API interactions and heuristics for selecting the appropriate API among a plethora of options. These steps eventually lead to the generation of API calls that are both syntactically and semantically aligned with the LLM's understanding of a given task. Moreover, we review existing frameworks and tools that facilitate these processes and highlight the gaps in current attempts. In this direction, we propose an on-device architecture that aims to exploit the functionality of carry-on devices by using small models from the Hugging Face community. We examine the effectiveness of these approaches on real-world applications of various domains, including the generation of a piano sheet. Through an extensive analysis of the literature and available technologies, this thesis aims to set a compass for researchers and practitioners to harness the full potential of LLMs augmented with external tool capabilities, thus paving the way for more autonomous, robust, and context-aware AI agents.