TL;DR
This paper introduces TIPO, a novel preference optimization method for mobile GUI agents that enhances privacy personalization by emphasizing privacy-related actions, leading to improved alignment and task success.
Contribution
The paper proposes TIPO, a new optimization approach that accounts for structural heterogeneity in trajectories to better personalize privacy preferences in mobile GUI agents.
Findings
TIPO improves persona alignment and distinction.
TIPO achieves 65.60% SR and 66.67% PD on privacy tasks.
TIPO outperforms existing methods in privacy-related GUI tasks.
Abstract
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
