HOT-POT: Optimal Transport for Sparse Stereo Matching

TL;DR

This paper introduces HOT-POT, a novel optimal transport-based method for sparse stereo matching that leverages camera geometry constraints to improve accuracy and efficiency in applications like facial analysis.

Contribution

It proposes a new OT-based framework for unsupervised sparse stereo and object matching using geometric constraints, extending to hierarchical OT for object matching.

Findings

Efficient algorithms for sparse feature and object matching.

Improved accuracy in facial landmark matching.

Demonstrated effectiveness in numerical experiments.

Abstract

Stereo vision between images faces a range of challenges, including occlusions, motion, and camera distortions, across applications in autonomous driving, robotics, and face analysis. Due to parameter sensitivity, further complications arise for stereo matching with sparse features, such as facial landmarks. To overcome this ill-posedness and enable unsupervised sparse matching, we consider line constraints of the camera geometry from an optimal transport (OT) viewpoint. Formulating camera-projected points as (half)lines, we propose the use of the classical epipolar distance as well as a 3D ray distance to quantify matching quality. Employing these distances as a cost function of a (partial) OT problem, we arrive at efficiently solvable assignment problems. Moreover, we extend our approach to unsupervised object matching by formulating it as a hierarchical OT problem. The resulting algorithms allow for efficient feature and object matching, as demonstrated in our numerical experiments. Here, we focus on applications in facial analysis, where we aim to match distinct landmarking conventions.