RESfM: Robust Deep Equivariant Structure from Motion

TL;DR

RESfM introduces a deep learning architecture for Structure from Motion that effectively handles outliers, improving accuracy in large, realistic image collections by combining equivariant classification and robust optimization.

Contribution

It presents a novel deep equivariant architecture with outlier classification and robust bundle adjustment for more realistic SfM scenarios.

Findings

Achieves state-of-the-art accuracy in large, outlier-prone datasets.

Outperforms existing deep SfM methods in realistic settings.

Matches classical methods' performance in challenging conditions.

Abstract

Multiview Structure from Motion is a fundamental and challenging computer vision problem. A recent deep-based approach utilized matrix equivariant architectures for simultaneous recovery of camera pose and 3D scene structure from large image collections. That work, however, made the unrealistic assumption that the point tracks given as input are almost clean of outliers. Here, we propose an architecture suited to dealing with outliers by adding a multiview inlier/outlier classification module that respects the model equivariance and by utilizing a robust bundle adjustment step. Experiments demonstrate that our method can be applied successfully in realistic settings that include large image collections and point tracks extracted with common heuristics that include many outliers, achieving state-of-the-art accuracies in almost all runs, superior to existing deep-based methods and on-par with leading classical (non-deep) sequential and global methods.