Deep Probabilistic Feature-metric Tracking

TL;DR

This paper introduces a deep probabilistic framework for dense RGB-D image alignment that predicts pixel-wise features and uncertainties, enabling robust and accurate pose estimation in challenging conditions.

Contribution

It presents a novel CNN-based method to learn deep feature maps and uncertainty estimates for probabilistic image alignment, integrated into a differentiable optimization framework.

Findings

Achieves state-of-the-art results on TUM RGB-D dataset.

Demonstrates robustness and reliable convergence in challenging scenarios.

Effectively couples with ICP for enhanced performance.

Abstract

Dense image alignment from RGB-D images remains a critical issue for real-world applications, especially under challenging lighting conditions and in a wide baseline setting. In this paper, we propose a new framework to learn a pixel-wise deep feature map and a deep feature-metric uncertainty map predicted by a Convolutional Neural Network (CNN), which together formulate a deep probabilistic feature-metric residual of the two-view constraint that can be minimised using Gauss-Newton in a coarse-to-fine optimisation framework. Furthermore, our network predicts a deep initial pose for faster and more reliable convergence. The optimisation steps are differentiable and unrolled to train in an end-to-end fashion. Due to its probabilistic essence, our approach can easily couple with other residuals, where we show a combination with ICP. Experimental results demonstrate state-of-the-art performances on the TUM RGB-D dataset and the 3D rigid object tracking dataset. We further demonstrate our method's robustness and convergence qualitatively.