Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representations

TL;DR

This paper introduces a language-based action representation for robotic control that normalizes actions to reduce distribution shifts, improving transferability and generalization across different robots and tasks.

Contribution

It proposes a semantically grounded linguistic motion representation that disregards numeric scales, enhancing pretraining transferability and reducing modality gaps in robotic manipulation.

Findings

Significant improvement in generalization performance.

Enhanced transferability across diverse robotic platforms.

Reduced modality gap between action and vocabulary tokens.

Abstract

Recent end-to-end robotic manipulation research increasingly adopts architectures inspired by large language models to enable robust manipulation. However, a critical challenge arises from severe distribution shifts between robotic action data, primarily due to substantial numerical variations in action commands across diverse robotic platforms and tasks, hindering the effective transfer of pretrained knowledge. To address this limitation, we propose a semantically grounded linguistic representation to normalize actions for efficient pretraining. Unlike conventional discretized action representations that are sensitive to numerical scales, the motion representation specifically disregards numeric scale effects, emphasizing directionality instead. This abstraction mitigates distribution shifts, yielding a more generalizable pretraining representation. Moreover, using the motion representation narrows the feature distance between action tokens and standard vocabulary tokens, mitigating modality gaps. Multi-task experiments on two benchmarks demonstrate that the proposed method significantly improves generalization performance and transferability in robotic manipulation tasks.