Visual Geometry Grounded Deep Structure From Motion

TL;DR

This paper introduces VGGSfM, a fully differentiable deep learning pipeline for structure-from-motion that improves camera pose and 3D reconstruction accuracy by integrating end-to-end training with novel mechanisms.

Contribution

It presents a novel end-to-end differentiable SfM pipeline that replaces traditional non-differentiable steps with learnable modules, enabling more accurate and efficient 3D reconstruction.

Findings

Achieves state-of-the-art results on CO3D, IMC Phototourism, and ETH3D datasets.

Eliminates the need for pairwise keypoint matching through pixel-accurate tracking.

Simultaneously recovers all camera poses and triangulates 3D points in a differentiable framework.

Abstract

Structure-from-motion (SfM) is a long-standing problem in the computer vision community, which aims to reconstruct the camera poses and 3D structure of a scene from a set of unconstrained 2D images. Classical frameworks solve this problem in an incremental manner by detecting and matching keypoints, registering images, triangulating 3D points, and conducting bundle adjustment. Recent research efforts have predominantly revolved around harnessing the power of deep learning techniques to enhance specific elements (e.g., keypoint matching), but are still based on the original, non-differentiable pipeline. Instead, we propose a new deep pipeline VGGSfM, where each component is fully differentiable and thus can be trained in an end-to-end manner. To this end, we introduce new mechanisms and simplifications. First, we build on recent advances in deep 2D point tracking to extract reliable pixel-accurate tracks, which eliminates the need for chaining pairwise matches. Furthermore, we recover all cameras simultaneously based on the image and track features instead of gradually registering cameras. Finally, we optimise the cameras and triangulate 3D points via a differentiable bundle adjustment layer. We attain state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism, and ETH3D.