Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

TL;DR

This paper introduces AgentFuel, a tool for creating domain-specific, expressive evaluation datasets for timeseries data analysis agents, addressing current limitations in agent evaluation and guiding improvements.

Contribution

AgentFuel enables domain experts to quickly generate customized evals for timeseries data agents, filling expressivity gaps in existing evaluation methods.

Findings

Existing agents fail on stateful and incident-specific queries.

AgentFuel's benchmarks reveal key areas for framework improvement.

Using AgentFuel can enhance agent performance, as shown with GEPA.

Abstract

Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to "talk to your data" to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agents (both open-source and proprietary) on domain-specific data and query types, and find that they fail on stateful and incident-specific queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel's benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks are available at https://huggingface.co/datasets/RockfishData/TimeSeriesAgentEvals.