OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation

TL;DR

OccuBench is a comprehensive benchmark that evaluates AI agents across 100 real-world professional tasks in various industries using language environment simulators, revealing diverse capabilities and robustness challenges.

Contribution

This work introduces OccuBench, the first large-scale, multi-domain benchmark for assessing AI agents' performance and robustness in professional environments via simulated scenarios.

Findings

No single model excels across all industries, showing diverse occupational profiles.

Implicit faults are more challenging than explicit errors, requiring autonomous detection.

Larger, newer models with more reasoning improve performance significantly.

Abstract

AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.