DiffPoseNet: Direct Differentiable Camera Pose Estimation

TL;DR

DiffPoseNet introduces a novel end-to-end differentiable framework for camera pose estimation that leverages normal flow and cheirality constraints, improving robustness and cross-dataset generalization over traditional methods.

Contribution

The paper presents a differentiable cheirality layer enabling end-to-end learning of camera pose from normal flow, bypassing the need for optical flow or scene structure.

Findings

Outperforms state-of-the-art methods on multiple datasets

Demonstrates robustness to noise and generalization across datasets

Enables end-to-end training of camera pose estimation models

Abstract

Current deep neural network approaches for camera pose estimation rely on scene structure for 3D motion estimation, but this decreases the robustness and thereby makes cross-dataset generalization difficult. In contrast, classical approaches to structure from motion estimate 3D motion utilizing optical flow and then compute depth. Their accuracy, however, depends strongly on the quality of the optical flow. To avoid this issue, direct methods have been proposed, which separate 3D motion from depth estimation but compute 3D motion using only image gradients in the form of normal flow. In this paper, we introduce a network NFlowNet, for normal flow estimation which is used to enforce robust and direct constraints. In particular, normal flow is used to estimate relative camera pose based on the cheirality (depth positivity) constraint. We achieve this by formulating the optimization problem as a differentiable cheirality layer, which allows for end-to-end learning of camera pose. We perform extensive qualitative and quantitative evaluation of the proposed DiffPoseNet's sensitivity to noise and its generalization across datasets. We compare our approach to existing state-of-the-art methods on KITTI, TartanAir, and TUM-RGBD datasets.