BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents

TL;DR

BEAP-Agent introduces a DFS-based framework for GUI agents that enables effective backtracking and adaptive planning, improving task exploration and recovery in complex GUI environments.

Contribution

It presents a novel backtracking mechanism for GUI agents using a DFS-based approach with dynamic task tracking, filling a gap in systematic exploration methods.

Findings

Achieved 28.2% accuracy on OSWorld benchmark.

Supports long-range, multi-level state backtracking.

Enhances robustness of GUI agents in task execution.

Abstract

GUI agents are designed to automate repetitive tasks and enhance productivity. However, existing GUI agents struggle to recover once they follow an incorrect exploration path, often leading to task failure. In this work, we model GUI task execution as a DFS process and propose BEAP-Agent, a DFS-based framework that supports long-range, multi-level state backtracking with dynamic task tracking and updating. The framework consists of three collaborative components: Planner, Executor, and Tracker. Together, they enable effective task exploration and execution. BEAP-Agent fills the gap in systematic backtracking mechanisms for GUI agents, offering a systematic solution for long-horizon task exploration. We conducted a systematic evaluation on the OSWorld benchmark, where BEAP-Agent achieved an accuracy of 28.2%, validating the effectiveness of the proposed method.