Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks

TL;DR

Odysseys introduces a benchmark of 200 real-world, long-horizon web tasks to evaluate web agents' sustained multi-site reasoning and efficiency, addressing limitations of existing short-task benchmarks.

Contribution

The paper presents Odysseys, a new benchmark with a rubric-based evaluation for long-horizon web tasks, and assesses current models' performance and efficiency on these tasks.

Findings

Strongest models achieve only 44.5% success rate.

Rubric-based evaluation aligns better with human judgment.

Agents exhibit only 1.15% efficiency, indicating room for improvement.

Abstract

Existing web agent benchmarks have largely converged on short, single-site tasks that frontier models are approaching saturation on. However, real world web use consists of long-horizon, multi-site workflows. Common web navigation tasks, such as comparing products across different domains, planning trips across multiple services, or summarizing information from multiple search queries, require sustained context and cross-site reasoning over potentially hours of browsing. To capture and evaluate such behaviors, we introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet. We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics. We demonstrate that this yields higher agreement with humans and provides a more fine-grained signal than commonly used trajectory-level LLM-as-a-judge evaluation metrics. We tested several leading frontier models and find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements. Beyond task success, we argue that efficiency is a first-class concern for long-horizon agents. We introduce a Trajectory Efficiency metric (rubric score per step) and find that even frontier agents achieve only 1.15%, marking an evident need for agents that can succeed efficiently and not simply eventually. Odysseys isolates the critical evaluation of long-horizon proficiency in open-web environments, providing a realistic benchmark to measure progress towards computer-use agents that can potentially productively operate for hours. We release our tasks, evaluation scripts, and other results at https://odysseys-website.pages.dev