EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents

TL;DR

EntWorld introduces a comprehensive benchmark for evaluating AI agents in complex enterprise environments, emphasizing realistic workflows, strict logic, and state verification, revealing significant gaps in current model capabilities.

Contribution

The paper presents EntWorld, a large-scale, schema-grounded enterprise benchmark with a novel SQL-based verification system, addressing limitations of previous datasets and fostering enterprise-specific AI development.

Findings

State-of-the-art models achieve only 47.61% success rate on EntWorld.

EntWorld reveals a substantial gap between current AI agents and human performance in enterprise tasks.

The benchmark enables rigorous evaluation of domain-specific agent capabilities.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled agents to operate in open-ended web and operating system environments. However, existing benchmarks predominantly target consumer-oriented scenarios (e.g., e-commerce and travel booking), failing to capture the complexity and rigor of professional enterprise workflows. Enterprise systems pose distinct challenges, including high-density user interfaces, strict business logic constraints, and a strong reliance on precise, state-consistent information retrieval-settings in which current generalist agents often struggle. To address this gap, we introduce EntWorld, a large-scale benchmark consisting of 1,756 tasks across six representative enterprise domains, including customer relationship management (CRM), information technology infrastructure library (ITIL), and enterprise resource planning (ERP) systems. Unlike previous datasets that depend on fragile execution traces or extensive manual annotation, EntWorld adopts a schema-grounded task generation framework that directly reverse-engineers business logic from underlying database schemas, enabling the synthesis of realistic, long-horizon workflows. Moreover, we propose a SQL-based deterministic verification mechanism in building datasets that replaces ambiguous visual matching with rigorous state-transition validation. Experimental results demonstrate that state-of-the-art models (e.g., GPT-4.1) achieve 47.61% success rate on EntWorld, substantially lower than the human performance, highlighting a pronounced enterprise gap in current agentic capabilities and the necessity of developing domain-specific agents. We release EntWorld as a rigorous testbed to facilitate the development and evaluation of the next generation of enterprise-ready digital agents.