Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking

TL;DR

This paper introduces a multi-modality 6DoF object tracking method that combines visual appearance and geometric information, achieving high accuracy and efficiency in robotic manipulation tasks.

Contribution

The work extends previous geometry-based tracking by integrating visual appearance features, including local keypoints and global region analysis, for improved pose estimation.

Findings

Outperforms existing methods on YCB-Video and OPT datasets.

Runs at over 300 Hz, demonstrating high efficiency.

Effectively combines appearance and geometry for robust tracking.

Abstract

In many applications of advanced robotic manipulation, six degrees of freedom (6DoF) object pose estimates are continuously required. In this work, we develop a multi-modality tracker that fuses information from visual appearance and geometry to estimate object poses. The algorithm extends our previous method ICG, which uses geometry, to additionally consider surface appearance. In general, object surfaces contain local characteristics from text, graphics, and patterns, as well as global differences from distinct materials and colors. To incorporate this visual information, two modalities are developed. For local characteristics, keypoint features are used to minimize distances between points from keyframes and the current image. For global differences, a novel region approach is developed that considers multiple regions on the object surface. In addition, it allows the modeling of external geometries. Experiments on the YCB-Video and OPT datasets demonstrate that our approach ICG+ performs best on both datasets, outperforming both conventional and deep learning-based methods. At the same time, the algorithm is highly efficient and runs at more than 300 Hz. The source code of our tracker is publicly available.