AgentStudio: A Toolkit for Building General Virtual Agents

TL;DR

AgentStudio introduces a versatile toolkit with environments, tools, and benchmarks to advance the development and evaluation of general virtual agents capable of handling multimodal data and complex actions in open environments.

Contribution

It provides a lightweight, interactive platform with new datasets and benchmarks for assessing fundamental agent capabilities in GUI interactions, video learning, and success detection.

Findings

Established three new datasets: GroundUI, IDMBench, CriticBench.

Created an online task suite for benchmarking GUI and function calling.

Reorganized existing datasets to support agent evaluation.

Abstract

General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which limits agent development and evaluation in real-world settings. As a result, current evaluations lack in-depth analyses that decompose fundamental agent capabilities. We introduce AgentStudio, a trinity of environments, tools, and benchmarks to address these issues. AgentStudio provides a lightweight, interactive environment with highly generic observation and action spaces, e.g., video observations and GUI/API actions. It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos. Based on our environment and tools, we curate an online task suite that benchmarks both GUI interactions and function calling with efficient auto-evaluation. We also reorganize existing datasets and collect new ones using our tools to establish three datasets: GroundUI, IDMBench, and CriticBench. These datasets evaluate fundamental agent abilities, including GUI grounding, learning from videos, and success detection, pointing to the desiderata for robust, general, and open-ended virtual agents.