# Expert Sample Consensus Applied to Camera Re-Localization

## TL;DR

This paper introduces Expert Sample Consensus (ESAC), a novel method combining Differentiable RANSAC with a Mixture of Experts to improve camera re-localization accuracy and scalability in noisy, ambiguous data scenarios.

## Contribution

We propose ESAC, an end-to-end trainable framework that integrates DSAC with MoE, enhancing robustness and efficiency in camera re-localization tasks.

## Key findings

- ESAC outperforms existing methods in scalability and ambiguity handling.
- Demonstrated improved accuracy in synthetic and real-world datasets.
- Efficient joint training of ESAC enhances model performance.

## Abstract

Fitting model parameters to a set of noisy data points is a common problem in computer vision. In this work, we fit the 6D camera pose to a set of noisy correspondences between the 2D input image and a known 3D environment. We estimate these correspondences from the image using a neural network. Since the correspondences often contain outliers, we utilize a robust estimator such as Random Sample Consensus (RANSAC) or Differentiable RANSAC (DSAC) to fit the pose parameters. When the problem domain, e.g. the space of all 2D-3D correspondences, is large or ambiguous, a single network does not cover the domain well. Mixture of Experts (MoE) is a popular strategy to divide a problem domain among an ensemble of specialized networks, so called experts, where a gating network decides which expert is responsible for a given input. In this work, we introduce Expert Sample Consensus (ESAC), which integrates DSAC in a MoE. Our main technical contribution is an efficient method to train ESAC jointly and end-to-end. We demonstrate experimentally that ESAC handles two real-world problems better than competing methods, i.e. scalability and ambiguity. We apply ESAC to fitting simple geometric models to synthetic images, and to camera re-localization for difficult, real datasets.