Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents

TL;DR

This paper presents ACuRL, an autonomous curriculum reinforcement learning framework that enables computer-use agents to adapt continually to diverse environments without human-labeled data, achieving significant performance improvements.

Contribution

It introduces a novel autonomous curriculum learning approach with a robust automatic evaluator, facilitating environment-specific adaptation without human supervision.

Findings

Achieves 3-29% performance gains in target environments.

Maintains robustness with 93% agreement between CUAJudge and human judgments.

Effectively mitigates performance degradation under environment changes.

Abstract

Real-world digital environments are highly diverse and dynamic. These characteristics cause agents to frequently encounter unseen environments and distribution shifts, making continual learning in such environments essential for computer-use agents (CUAs). However, a key challenge lies in obtaining high-quality and environment-grounded training data without relying on costly human annotation. In this work, we introduce ACuRL, an Autonomous Curriculum Reinforcement Learning framework that continually adapts agents to specific environments with zero human data. The agent first explores an environment to acquire initial experiences. During subsequent iterative training, a curriculum task generator leverages these experiences together with feedback from the previous iteration to synthesize new tasks tailored for the agent's current capabilities. To provide reliable reward signals, we introduce CUAJudge, a robust automatic evaluator for CUAs that achieves 93% agreement with human judgments. Empirically, our method effectively enables both intra-environment and cross-environment continual learning, yielding 3-29% absolute performance gains on the target environments without catastrophic forgetting on others. We also show that it can mitigate performance degradation under environment changes (e.g., version updates, platform migration, and resolution shifts). Further analyses show highly sparse updates (e.g., only 20% parameters), which helps explain the effective and robust adaptation.