OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks

TL;DR

OS-Marathon introduces a comprehensive benchmark for evaluating computer-use agents on long, repetitive workflows, and proposes a few-shot learning method to improve their performance on unseen data.

Contribution

The paper establishes OS-Marathon, a new benchmark with 242 tasks, and presents a cost-effective few-shot approach for training agents on long-horizon repetitive tasks.

Findings

SOTA agents face significant challenges on long-horizon tasks.

Few-shot demonstration methods improve agent performance.

Benchmark highlights the complexity of real-world repetitive workflows.

Abstract

Long-horizon, repetitive workflows are common in professional settings, such as processing expense reports from receipts and entering student grades from exam papers. These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process. However, they are ideal for Computer-Use Agents (CUAs) due to their structured, recurring sub-workflows with logic that can be systematically learned. Identifying the absence of an evaluation benchmark as a primary bottleneck, we establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents. We then introduce a cost-effective method to construct a condensed demonstration using only few-shot examples to teach agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections. Extensive experiments demonstrate both the inherent challenges of these tasks and the effectiveness of our proposed method. Project website: https://os-marathon.github.io/.