NeFSAC: Neurally Filtered Minimal Samples

TL;DR

NeFSAC is a neural filtering algorithm that improves RANSAC-based pose estimation by early recognition of invalid samples, leading to significant speed-ups and often more accurate results across diverse datasets.

Contribution

We introduce NeFSAC, a lightweight neural filtering method that predicts the validity of minimal samples for pose estimation, enhancing RANSAC efficiency and robustness.

Findings

NeFSAC achieves up to tenfold speed-up in RANSAC pipelines.

It maintains high accuracy even across different domain datasets.

NeFSAC outperforms traditional methods in challenging scenarios.

Abstract

Since RANSAC, a great deal of research has been devoted to improving both its accuracy and run-time. Still, only a few methods aim at recognizing invalid minimal samples early, before the often expensive model estimation and quality calculation are done. To this end, we propose NeFSAC, an efficient algorithm for neural filtering of motion-inconsistent and poorly-conditioned minimal samples. We train NeFSAC to predict the probability of a minimal sample leading to an accurate relative pose, only based on the pixel coordinates of the image correspondences. Our neural filtering model learns typical motion patterns of samples which lead to unstable poses, and regularities in the possible motions to favour well-conditioned and likely-correct samples. The novel lightweight architecture implements the main invariants of minimal samples for pose estimation, and a novel training scheme addresses the problem of extreme class imbalance. NeFSAC can be plugged into any existing RANSAC-based pipeline. We integrate it into USAC and show that it consistently provides strong speed-ups even under extreme train-test domain gaps - for example, the model trained for the autonomous driving scenario works on PhotoTourism too. We tested NeFSAC on more than 100k image pairs from three publicly available real-world datasets and found that it leads to one order of magnitude speed-up, while often finding more accurate results than USAC alone. The source code is available at https://github.com/cavalli1234/NeFSAC.