ColorAgent: Building A Robust, Personalized, and Interactive OS Agent

TL;DR

ColorAgent is a novel OS agent that combines reinforcement learning, multi-agent frameworks, and personalized interaction to enable robust, long-horizon, and user-centric operations, achieving state-of-the-art success rates on benchmark tests.

Contribution

The paper introduces ColorAgent, a comprehensive OS agent with enhanced capabilities for long-term interaction and personalization, utilizing step-wise reinforcement learning and multi-agent systems.

Findings

Achieved 77.2% success on AndroidWorld benchmark.

Achieved 50.7% success on AndroidLab benchmark.

Proposed new directions for evaluation and collaboration in OS agents.

Abstract

With the advancements in hardware, software, and large language model technologies, the interaction between humans and operating systems has evolved from the command-line interface to the rapidly emerging AI agent interactions. Building an operating system (OS) agent capable of executing user instructions and faithfully following user desires is becoming a reality. In this technical report, we present ColorAgent, an OS agent designed to engage in long-horizon, robust interactions with the environment while also enabling personalized and proactive user interaction. To enable long-horizon interactions with the environment, we enhance the model's capabilities through step-wise reinforcement learning and self-evolving training, while also developing a tailored multi-agent framework that ensures generality, consistency, and robustness. In terms of user interaction, we explore personalized user intent recognition and proactive engagement, positioning the OS agent not merely as an automation tool but as a warm, collaborative partner. We evaluate ColorAgent on the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2% and 50.7%, respectively, establishing a new state of the art. Nonetheless, we note that current benchmarks are insufficient for a comprehensive evaluation of OS agents and propose further exploring directions in future work, particularly in the areas of evaluation paradigms, agent collaboration, and security.