Evaluating LLMs on Sequential API Call Through Automated Test Generation

TL;DR

This paper introduces StateGen, an automated framework for generating and evaluating complex sequential API call tasks for LLMs, addressing limitations of existing benchmarks by providing diverse, verifiable, and realistic test cases.

Contribution

The paper presents StateGen and StateEval, novel tools for automated, realistic, and verifiable API interaction testing of LLMs, filling a gap in current evaluation methods.

Findings

StateGen effectively generates challenging API-oriented tasks.

StateEval provides a diverse benchmark with 120 verified test cases.

Results highlight current LLM limitations in API integration.

Abstract

By integrating tools from external APIs, Large Language Models (LLMs) have expanded their promising capabilities in a diverse spectrum of complex real-world tasks. However, testing, evaluation, and analysis of LLM tool use remain in their early stages. Most existing benchmarks rely on manually collected test cases, many of which cannot be automatically checked for semantic correctness and instead depend on static methods such as string matching. Additionally, these benchmarks often overlook the complex interactions that occur between sequential API calls, which are common in real-world applications. To fill the gap, in this paper, we introduce StateGen, an automated framework designed to generate diverse coding tasks involving sequential API interactions. StateGen combines state-machine-based API constraint solving and validation, energy-based sampling, and control-flow injection to generate executable programs. These programs are then translated into human-like natural language task descriptions through a collaboration of two LLM agents. Utilizing StateGen, we construct StateEval, a benchmark encompassing 120 verified test cases spanning across three representative scenarios: Session Service, Tensor Operation, and ElevenLabs MCP. Experimental results confirm that StateGen can effectively generate challenging and realistic API-oriented tasks, highlighting areas for improvement in current LLMs incorporating APIs.We make our framework and benchmark publicly available to support future research.