LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization

TL;DR

LPO is a novel method that improves GUI agent interaction accuracy by optimizing location preferences using information entropy and dynamic rewards, outperforming existing methods.

Contribution

Introduces Location Preference Optimization (LPO), a new approach leveraging locational data and information entropy to enhance GUI interaction precision.

Findings

LPO achieves state-of-the-art results on offline benchmarks.

LPO significantly improves real-world online interaction accuracy.

Supported by extensive experiments demonstrating superior performance.

Abstract

The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/jqtangust/LPO.