DPO Learning with LLMs-Judge Signal for Computer Use Agents

TL;DR

This paper introduces a lightweight, privacy-preserving vision-language model for GUI agents that uses an LLM-based judge to automatically evaluate training data, resulting in improved local performance on GUI tasks.

Contribution

The work presents a novel LLM-as-Judge framework for training compact GUI agents without human labels, enabling efficient, private, and scalable local inference.

Findings

Outperforms existing baselines on OS-World benchmark

Enables privacy-preserving local GUI agent operation

Demonstrates effective reinforcement learning data filtering

Abstract

Computer use agents (CUA) are systems that automatically interact with graphical user interfaces (GUIs) to complete tasks. CUA have made significant progress with the advent of large vision-language models (VLMs). However, these agents typically rely on cloud-based inference with substantial compute demands, raising critical privacy and scalability concerns, especially when operating on personal devices. In this work, we take a step toward privacy-preserving and resource-efficient agents by developing a lightweight vision-language model that runs entirely on local machines. To train this compact agent, we introduce an LLM-as-Judge framework that automatically evaluates and filters synthetic interaction trajectories, producing high-quality data for reinforcement learning without human annotation. Experiments on the OS-World benchmark demonstrate that our fine-tuned local model outperforms existing baselines, highlighting a promising path toward private, efficient, and generalizable GUI agents.