3D Implicit Transporter for Temporally Consistent Keypoint Discovery

TL;DR

This paper introduces the first 3D implicit Transporter model that ensures spatio-temporal consistency in keypoint detection for 3D objects, improving 3D manipulation tasks.

Contribution

It develops a novel 3D Transporter leveraging hybrid representations, cross attention, and implicit reconstruction, addressing the limitations of 2D methods for 3D data.

Findings

Learned keypoints are spatio-temporally consistent.

The method outperforms existing approaches in 3D object manipulation.

Codes are publicly available for reproducibility.

Abstract

Keypoint-based representation has proven advantageous in various visual and robotic tasks. However, the existing 2D and 3D methods for detecting keypoints mainly rely on geometric consistency to achieve spatial alignment, neglecting temporal consistency. To address this issue, the Transporter method was introduced for 2D data, which reconstructs the target frame from the source frame to incorporate both spatial and temporal information. However, the direct application of the Transporter to 3D point clouds is infeasible due to their structural differences from 2D images. Thus, we propose the first 3D version of the Transporter, which leverages hybrid 3D representation, cross attention, and implicit reconstruction. We apply this new learning system on 3D articulated objects and nonrigid animals (humans and rodents) and show that learned keypoints are spatio-temporally consistent. Additionally, we propose a closed-loop control strategy that utilizes the learned keypoints for 3D object manipulation and demonstrate its superior performance. Codes are available at https://github.com/zhongcl-thu/3D-Implicit-Transporter.