# Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses

## TL;DR

Neural-Guided RANSAC (NG-RANSAC) enhances the classic RANSAC algorithm by learning to guide hypothesis sampling using neural networks, resulting in improved performance on various computer vision tasks.

## Contribution

We introduce NG-RANSAC, a novel method that learns to guide hypothesis sampling in RANSAC, optimizing task-specific loss functions and enabling self-supervised training.

## Key findings

- NG-RANSAC outperforms traditional RANSAC in multiple vision tasks.
- Neural guidance improves hypothesis selection efficiency.
- Self-supervised training of neural guidance is effective.

## Abstract

We present Neural-Guided RANSAC (NG-RANSAC), an extension to the classic RANSAC algorithm from robust optimization. NG-RANSAC uses prior information to improve model hypothesis search, increasing the chance of finding outlier-free minimal sets. Previous works use heuristic side-information like hand-crafted descriptor distance to guide hypothesis search. In contrast, we learn hypothesis search in a principled fashion that lets us optimize an arbitrary task loss during training, leading to large improvements on classic computer vision tasks. We present two further extensions to NG-RANSAC. Firstly, using the inlier count itself as training signal allows us to train neural guidance in a self-supervised fashion. Secondly, we combine neural guidance with differentiable RANSAC to build neural networks which focus on certain parts of the input data and make the output predictions as good as possible. We evaluate NG-RANSAC on a wide array of computer vision tasks, namely estimation of epipolar geometry, horizon line estimation and camera re-localization. We achieve superior or competitive results compared to state-of-the-art robust estimators, including very recent, learned ones.