AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent
Neeraj Anand, Rishabh Jain, Sohan Patnaik, Balaji Krishnamurthy, Mausoom Sarkar

TL;DR
AFRAgent is a compact multimodal GUI automation model that uses adaptive feature renormalization to improve widget identification and task execution, achieving state-of-the-art results on smartphone automation benchmarks.
Contribution
This work introduces AFRAgent, a novel, smaller multimodal architecture with adaptive feature renormalization that enhances image embeddings for GUI automation tasks.
Findings
Achieves superior performance on Meta-GUI and AITW benchmarks.
Less than one-fourth the size of comparable models.
Establishes new state-of-the-art in smartphone automation.
Abstract
There is a growing demand for mobile user interface (UI) automation, driven by its broad applications across industries. With the advent of visual language models (VLMs), GUI automation has progressed from generating text-based instructions for humans to autonomously executing tasks, thus optimizing automation workflows. Recent approaches leverage VLMs for this problem due to their ability to 1) process on-screen content directly, 2) remain independent of device-specific APIs by utilizing human actions (e.g., clicks, typing), and 3) apply real-world contextual knowledge for task understanding. However, these models often have trouble accurately identifying widgets and determining actions due to limited spatial information in vision encoder features. Additionally, top-performing models are often large, requiring extensive training and resulting in inference delays. In this work, we…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications
