Learning Structure-from-Motion with Graph Attention Networks

TL;DR

This paper introduces a novel graph attention network approach for learning Structure-from-Motion, replacing traditional sub-problems with a learned model that achieves faster inference and outperforms existing learning-based methods.

Contribution

It presents a new graph neural network model that directly predicts camera poses and 3D points from 2D keypoints, improving speed and accuracy over prior learning-based SfM methods.

Findings

Outperforms existing learning-based SfM methods.

Achieves faster inference than traditional methods.

Challenges COLMAP with lower runtime.

Abstract

In this paper we tackle the problem of learning Structure-from-Motion (SfM) through the use of graph attention networks. SfM is a classic computer vision problem that is solved though iterative minimization of reprojection errors, referred to as Bundle Adjustment (BA), starting from a good initialization. In order to obtain a good enough initialization to BA, conventional methods rely on a sequence of sub-problems (such as pairwise pose estimation, pose averaging or triangulation) which provide an initial solution that can then be refined using BA. In this work we replace these sub-problems by learning a model that takes as input the 2D keypoints detected across multiple views, and outputs the corresponding camera poses and 3D keypoint coordinates. Our model takes advantage of graph neural networks to learn SfM-specific primitives, and we show that it can be used for fast inference of the reconstruction for new and unseen sequences. The experimental results show that the proposed model outperforms competing learning-based methods, and challenges COLMAP while having lower runtime. Our code is available at https://github.com/lucasbrynte/gasfm/.