Structural Action Transformer for 3D Dexterous Manipulation

TL;DR

This paper introduces the Structural Action Transformer (SAT), a novel 3D manipulation policy that uses a structural-centric approach to enable effective cross-embodiment skill transfer for high-DoF robotic hands.

Contribution

The paper proposes a structural-centric action representation and an Embodied Joint Codebook to improve cross-embodiment transfer and sample efficiency in 3D dexterous manipulation.

Findings

Outperforms baseline methods in simulation and real-world tasks

Demonstrates superior sample efficiency in learning from heterogeneous datasets

Enables effective transfer of skills across different robotic hand embodiments

Abstract

Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.