SEAL: Suite for Evaluating API-use of LLMs

TL;DR

SEAL is a comprehensive, standardized testbed for evaluating large language models' ability to use external APIs effectively, addressing real-world challenges like API instability and multi-step reasoning.

Contribution

The paper introduces SEAL, a novel end-to-end evaluation framework that standardizes API-use benchmarks, incorporates an API simulator, and enhances evaluation reliability for LLMs.

Findings

Provides a deterministic evaluation environment using GPT-4-powered API simulation.

Addresses API instability issues with caching mechanisms.

Offers a comprehensive pipeline covering retrieval, API calls, and responses.

Abstract

Large language models (LLMs) have limitations in handling tasks that require real-time access to external APIs. While several benchmarks like ToolBench and APIGen have been developed to assess LLMs' API-use capabilities, they often suffer from issues such as lack of generalizability, limited multi-step reasoning coverage, and instability due to real-time API fluctuations. In this paper, we introduce SEAL, an end-to-end testbed designed to evaluate LLMs in real-world API usage. SEAL standardizes existing benchmarks, integrates an agent system for testing API retrieval and planning, and addresses the instability of real-time APIs by introducing a GPT-4-powered API simulator with caching for deterministic evaluations. Our testbed provides a comprehensive evaluation pipeline that covers API retrieval, API calls, and final responses, offering a reliable framework for structured performance comparison in diverse real-world scenarios. SEAL is publicly available, with ongoing updates for new benchmarks.