iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception
Sarthak Mehrotra, Sairam V C Rebbapragada, Mani Hemanth Reddy Bonthu, Vineeth N Balasubramanian

TL;DR
iSHIFT is a lightweight multimodal GUI agent that adaptively switches between detailed and global visual reasoning modes, achieving high accuracy and efficiency in complex GUI tasks with a compact model.
Contribution
This work introduces iSHIFT, a novel lightweight GUI agent combining implicit reasoning with adaptive perception control, enabling flexible, efficient, and precise interactions.
Findings
iSHIFT matches state-of-the-art performance on benchmark datasets.
It operates with a compact 2.5B parameter model.
The adaptive switching improves both efficiency and accuracy.
Abstract
Multimodal Large Language Models (MLLMs) show strong potential for interpreting and interacting with complex, pixel-rich Graphical User Interface (GUI) environments. However, building agents that are both efficient for high-level tasks and precise for fine-grained interactions remains challenging. GUI agents must perform routine actions efficiently while also handling tasks that demand exact visual grounding, yet existing approaches struggle when accuracy depends on identifying specific interface elements. These MLLMs also remain large and cannot adapt their reasoning depth to the task at hand. In this work, we introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking (implicit chain-of-thought) with a perception control module. iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Generative Adversarial Networks and Image Synthesis · Explainable Artificial Intelligence (XAI)
