Learning API Functionality from In-Context Demonstrations for Tool-based Agents

TL;DR

This paper introduces a novel approach for API functionality learning directly from in-context demonstrations, enabling tool-based agents to operate effectively without relying on documentation, and analyzes factors affecting success rates.

Contribution

It proposes a new paradigm for learning API functions from demonstrations, bypassing documentation, and provides extensive analysis on factors influencing task success in this setting.

Findings

Explicit function calls improve accuracy

Natural language critiques enhance success rates

Learning from demonstrations remains challenging for state-of-the-art LLMs

Abstract

Digital tool-based agents, powered by Large Language Models (LLMs), that invoke external Application Programming Interfaces (APIs) often rely on documentation to understand API functionality. However, such documentation is frequently missing, outdated, privatized, or inconsistent-hindering the development of reliable, general-purpose agents. In this work, we propose a new research direction: learning of API functionality directly from in-context demonstrations. This task is a new paradigm applicable in scenarios without documentation. Using API benchmarks, we collect demonstrations from both expert agents and from self-exploration. To understand what information demonstrations must convey for successful task completion, we extensively study how the number of demonstrations and the use of LLM-generated summaries and evaluations affect the task success rate of the API-based agent. Our experiments across 3 datasets and 6 models show that learning functionality from in-context demonstrations remains a non-trivial challenge, even for state-of-the-art LLMs. We find that providing explicit function calls and natural language critiques significantly improves the agent's task success rate due to more accurate parameter filling. We analyze failure modes, identify sources of error, and highlight key open challenges for future work in documentation-free, self-improving, API-based agents.