ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

TL;DR

ToolTok introduces a multi-step pathfinding approach for GUI agents, leveraging tool tokenization and semantic grounding to improve generalization and efficiency with limited data, outperforming larger models on benchmarks.

Contribution

It proposes a novel tool tokenization paradigm with semantic anchoring and curriculum learning, enabling efficient training and strong generalization in GUI agents.

Findings

Outperforms comparable-scale models on multiple benchmarks.

Achieves competitive results with a much larger model using less than 1% of training data.

Demonstrates strong generalization to unseen scenarios.

Abstract

Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training & inference code is open-source at https://github.com/ZephinueCode/ToolTok.