ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li

TL;DR
ToolTok introduces a multi-step pathfinding approach for GUI agents, leveraging tool tokenization and semantic grounding to improve generalization and efficiency with limited data, outperforming larger models on benchmarks.
Contribution
It proposes a novel tool tokenization paradigm with semantic anchoring and curriculum learning, enabling efficient training and strong generalization in GUI agents.
Findings
Outperforms comparable-scale models on multiple benchmarks.
Achieves competitive results with a much larger model using less than 1% of training data.
Demonstrates strong generalization to unseen scenarios.
Abstract
Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Neural Network Applications · Generative Adversarial Networks and Image Synthesis
