BA-Net: Dense Bundle Adjustment Network

TL;DR

BA-Net introduces a differentiable deep learning architecture that integrates multi-view geometry constraints with feature learning to improve dense structure-from-motion, enabling end-to-end training for accurate depth reconstruction.

Contribution

It presents a novel deep network combining feature-metric bundle adjustment with a new depth parameterization for dense SfM, integrating domain knowledge with learning.

Findings

Successfully recovers dense per-pixel depth from real data

Outperforms existing methods on large-scale SfM benchmarks

Demonstrates effective integration of geometry constraints with deep learning

Abstract

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.