ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution

TL;DR

ASPERA is a framework that evaluates large language models' ability to generate complex multi-step action programs for digital assistants, addressing data and robustness challenges in task execution.

Contribution

We introduce ASPERA, a novel simulation environment and dataset for assessing LLMs' capacity to generate complex, library-dependent action programs for digital assistants.

Findings

LLMs struggle with program generation grounded in custom libraries.

ASPERA's dataset reveals significant challenges in dependency-based code generation.

Framework improves evaluation robustness for complex task execution.

Abstract

This work evaluates the potential of large language models (LLMs) to power digital assistants capable of complex action execution. These assistants rely on pre-trained programming knowledge to execute multi-step goals by composing objects and functions defined in assistant libraries into action execution programs. To achieve this, we develop ASPERA, a framework comprising an assistant library simulation and a human-assisted LLM data generation engine. Our engine allows developers to guide LLM generation of high-quality tasks consisting of complex user queries, simulation state and corresponding validation programs, tackling data availability and evaluation robustness challenges. Alongside the framework we release Asper-Bench, an evaluation dataset of 250 challenging tasks generated using ASPERA, which we use to show that program generation grounded in custom assistant libraries is a significant challenge to LLMs compared to dependency-free code generation.