Mimic Intent, Not Just Trajectories

TL;DR

This paper introduces MINT, a novel imitation learning approach that disentangles intent from execution using multi-scale spectral tokenization, enabling better transfer, generalization, and efficiency in manipulation tasks.

Contribution

MINT employs multi-scale frequency-space tokenization to explicitly separate intent from execution, improving transferability and generalization in imitation learning.

Findings

Achieves state-of-the-art success rates on manipulation benchmarks.

Enables effective one-shot skill transfer by injecting intent tokens.

Demonstrates robustness and efficiency in real robot experiments.

Abstract

While imitation learning (IL) has achieved impressive success in dexterous manipulation through generative modeling and pretraining, state-of-the-art approaches like Vision-Language-Action (VLA) models still struggle with adaptation to environmental changes and skill transfer. We argue this stems from mimicking raw trajectories without understanding the underlying intent. To address this, we propose explicitly disentangling behavior intent from execution details in end-2-end IL: Mimic Intent, Not just Trajectories(MINT). We achieve this via multi-scale frequency-space tokenization, which enforces a spectral decomposition of action chunk representation. We learn action tokens with a multi-scale coarse-to-fine structure, and force the coarsest token to capture low-frequency global structure and finer tokens to encode high-frequency details. This yields an abstract Intent token that facilitates planning and transfer, and multi-scale Execution tokens that enable precise adaptation to environmental dynamics. Building on this hierarchy, our policy generates trajectories through next-scale autoregression, performing progressive intent-to-execution reasoning, thus boosting learning efficiency and generalization. Crucially, this disentanglement enables one-shot transfer of skills, by simply injecting the Intent token from a demonstration into the autoregressive generation process. Experiments on several manipulation benchmarks and on a real robot demonstrate state-of-the-art success rates, superior inference efficiency, robust generalization against disturbances, and effective one-shot transfer.