From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

TL;DR

This paper introduces BEPA, a novel method that leverages expert trajectories to improve end-to-end GUI agents by aligning static expert data with the learner's policy through a bi-level assimilation process, enhancing performance on benchmark tasks.

Contribution

BEPA presents a new bi-level approach to incorporate expert trajectories into reinforcement learning for GUI agents, addressing distribution mismatch and improving success rates on benchmark environments.

Findings

BEPA increases success rates on OSWorld-Verified from 22.87% to 32.13%.

BEPA improves performance on MMBench-GUI and Online-Mind2Web benchmarks.

The method effectively aligns expert data with the learner's policy, enabling better utilization of limited expert trajectories.

Abstract

Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at: https://github.com/LEON-gittech/Verl_GUI.git