Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

TL;DR

Iris is a novel visual agent that improves GUI interaction by dynamically focusing on dense regions and self-refining its understanding, achieving state-of-the-art results with minimal annotated data.

Contribution

The paper introduces Iris, featuring Information-Sensitive Cropping and Self-Refining Dual Learning, which enhance visual perception and task performance in GUI environments without extensive annotations.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Outperforms methods using ten times more training data.

Significantly improves downstream web and OS agent tasks.

Abstract

Digital agents are increasingly employed to automate tasks in interactive digital environments such as web pages, software applications, and operating systems. While text-based agents built on Large Language Models (LLMs) often require frequent updates due to platform-specific APIs, visual agents leveraging Multimodal Large Language Models (MLLMs) offer enhanced adaptability by interacting directly with Graphical User Interfaces (GUIs). However, these agents face significant challenges in visual perception, particularly when handling high-resolution, visually complex digital environments. This paper introduces Iris, a foundational visual agent that addresses these challenges through two key innovations: Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL). ISC dynamically identifies and prioritizes visually dense regions using a edge detection algorithm, enabling efficient processing by allocating more computational resources to areas with higher information density. SRDL enhances the agent's ability to handle complex tasks by leveraging a dual-learning loop, where improvements in referring (describing UI elements) reinforce grounding (locating elements) and vice versa, all without requiring additional annotated data. Empirical evaluations demonstrate that Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations, outperforming methods using 10x more training data. These improvements further translate to significant gains in both web and OS agent downstream tasks.