MinBackProp -- Backpropagating through Minimal Solvers

TL;DR

This paper introduces a stable, fast method for backpropagating through minimal problem solvers in neural networks using the Implicit Function Theorem, improving efficiency and stability over traditional approaches.

Contribution

The authors propose using the Implicit Function Theorem for backpropagation through minimal solvers, offering a simple, fast, and stable alternative to existing methods.

Findings

100% stability in backpropagation through minimal solvers

10 times faster than autograd-based methods

Effective in real-world image matching applications

Abstract

We present an approach to backpropagating through minimal problem solvers in end-to-end neural network training. Traditional methods relying on manually constructed formulas, finite differences, and autograd are laborious, approximate, and unstable for complex minimal problem solvers. We show that using the Implicit function theorem (IFT) to calculate derivatives to backpropagate through the solution of a minimal problem solver is simple, fast, and stable. We compare our approach to (i) using the standard autograd on minimal problem solvers and relate it to existing backpropagation formulas through SVD-based and Eig-based solvers and (ii) implementing the backprop with an existing PyTorch Deep Declarative Networks (DDN) framework. We demonstrate our technique on a toy example of training outlier-rejection weights for 3D point registration and on a real application of training an outlier-rejection and RANSAC sampling network in image matching. Our method provides $100\%$ stability and is 10 times faster compared to autograd, which is unstable and slow, and compared to DDN, which is stable but also slow.