Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation

TL;DR

Act3D introduces a 3D feature field transformer for robotic manipulation that adaptively samples high-resolution 3D points, achieving state-of-the-art results with less computation by leveraging 2D pre-trained features and relative attention.

Contribution

This paper presents a novel 3D manipulation policy transformer that efficiently combines 2D pre-trained features with adaptive 3D sampling for improved spatial reasoning.

Findings

Achieves 10% improvement over previous 2D policies on RL-Bench tasks.

Attains 22% higher success rate with 3x less compute than previous 3D methods.

Demonstrates the effectiveness of relative spatial attention and large-scale pre-trained backbones.

Abstract

3D perceptual representations are well suited for robot manipulation as they easily encode occlusions and simplify spatial reasoning. Many manipulation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that are computationally expensive to process. As a result, most manipulation policies operate directly in 2D, foregoing 3D inductive biases. In this paper, we introduce Act3D, a manipulation policy transformer that represents the robot's workspace using a 3D feature field with adaptive resolutions dependent on the task at hand. The model lifts 2D pre-trained features to 3D using sensed depth, and attends to them to compute features for sampled 3D points. It samples 3D point grids in a coarse to fine manner, featurizes them using relative-position attention, and selects where to focus the next round of point sampling. In this way, it efficiently computes 3D action maps of high spatial resolution. Act3D sets a new state-of-the-art in RL-Bench, an established manipulation benchmark, where it achieves 10% absolute improvement over the previous SOTA 2D multi-view policy on 74 RLBench tasks and 22% absolute improvement with 3x less compute over the previous SOTA 3D policy. We quantify the importance of relative spatial attention, large-scale vision-language pre-trained 2D backbones, and weight tying across coarse-to-fine attentions in ablative experiments. Code and videos are available on our project website: https://act3d.github.io/.