Emergent Outlier View Rejection in Visual Geometry Grounded Transformers

TL;DR

This paper reveals that certain layers in existing feed-forward 3D reconstruction models inherently suppress outlier images, enabling effective noise filtering without additional training, thus improving in-the-wild 3D reconstruction.

Contribution

The study uncovers an implicit outlier rejection mechanism within a specific layer of VGGT, allowing noise filtering in 3D reconstruction without explicit outlier handling.

Findings

Identifies a layer with natural outlier suppression behavior.

Demonstrates effective outlier rejection without extra supervision.

Shows consistent performance across diverse datasets.

Abstract

Reliable 3D reconstruction from in-the-wild image collections is often hindered by "noisy" images-irrelevant inputs with little or no view overlap with others. While traditional Structure-from-Motion pipelines handle such cases through geometric verification and outlier rejection, feed-forward 3D reconstruction models lack these explicit mechanisms, leading to degraded performance under in-the-wild conditions. In this paper, we discover that the existing feed-forward reconstruction model, e.g., VGGT, despite lacking explicit outlier-rejection mechanisms or noise-aware training, can inherently distinguish distractor images. Through an in-depth analysis under varying proportions of synthetic distractors, we identify a specific layer that naturally exhibits outlier-suppressing behavior. Further probing reveals that this layer encodes discriminative internal representations that enable an effective noise-filtering capability, which we simply leverage to perform outlier-view rejection in feed-forward 3D reconstruction without any additional fine-tuning or supervision. Extensive experiments on both controlled and in-the-wild datasets demonstrate that this implicit filtering mechanism is consistent and generalizes well across diverse scenarios.