OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

TL;DR

OSCAR is a versatile agent that uses state-aware reasoning and re-planning to autonomously control desktop and mobile applications, translating instructions into Python code for precise GUI interactions, improving task automation.

Contribution

This work introduces OSCAR, a novel generalist agent that combines state machine reasoning and dynamic re-planning to enhance automation across diverse applications.

Findings

Effective in automating complex workflows

Significantly improves user productivity

Operates reliably across platforms

Abstract

Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR's effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.