Doppelgangers++: Improved Visual Disambiguation with Geometric 3D Features

TL;DR

Doppelgangers++ introduces a robust, scene-agnostic method for detecting visually similar but distinct surfaces to improve 3D reconstruction accuracy, leveraging a diversified dataset and a Transformer-based classifier with 3D-aware features.

Contribution

The paper presents a novel Transformer-based classifier using 3D-aware features and a diversified dataset to enhance visual disambiguation in 3D reconstruction tasks.

Findings

Significantly improves pairwise visual disambiguation accuracy.

Enhances 3D reconstruction quality across diverse real-world scenes.

Seamlessly integrates into existing SfM pipelines.

Abstract

Accurate 3D reconstruction is frequently hindered by visual aliasing, where visually similar but distinct surfaces (aka, doppelgangers), are incorrectly matched. These spurious matches distort the structure-from-motion (SfM) process, leading to misplaced model elements and reduced accuracy. Prior efforts addressed this with CNN classifiers trained on curated datasets, but these approaches struggle to generalize across diverse real-world scenes and can require extensive parameter tuning. In this work, we present Doppelgangers++, a method to enhance doppelganger detection and improve 3D reconstruction accuracy. Our contributions include a diversified training dataset that incorporates geo-tagged images from everyday scenes to expand robustness beyond landmark-based datasets. We further propose a Transformer-based classifier that leverages 3D-aware features from the MASt3R model, achieving superior precision and recall across both in-domain and out-of-domain tests. Doppelgangers++ integrates seamlessly into standard SfM and MASt3R-SfM pipelines, offering efficiency and adaptability across varied scenes. To evaluate SfM accuracy, we introduce an automated, geotag-based method for validating reconstructed models, eliminating the need for manual inspection. Through extensive experiments, we demonstrate that Doppelgangers++ significantly enhances pairwise visual disambiguation and improves 3D reconstruction quality in complex and diverse scenarios.