UFV-Splatter: Pose-Free Feed-Forward 3D Gaussian Splatting Adapted to Unfavorable Views

TL;DR

This paper introduces UFV-Splatter, a novel framework that enables pose-free, feed-forward 3D Gaussian Splatting models to effectively handle unfavorable views by leveraging priors, adaptation layers, and a new training strategy.

Contribution

It proposes a new adaptation framework with Gaussian modules and training strategies to extend 3D Gaussian Splatting models to unfavorable views without requiring pose annotations.

Findings

Effective handling of unfavorable views demonstrated on synthetic and real datasets.

Improved geometric consistency and view rendering accuracy.

Compatibility with pretrained pose-free 3D Gaussian Splatting models.

Abstract

This paper presents a pose-free, feed-forward 3D Gaussian Splatting (3DGS) framework designed to handle unfavorable input views. A common rendering setup for training feed-forward approaches places a 3D object at the world origin and renders it from cameras pointed toward the origin -- i.e., from favorable views, limiting the applicability of these models to real-world scenarios involving varying and unknown camera poses. To overcome this limitation, we introduce a novel adaptation framework that enables pretrained pose-free feed-forward 3DGS models to handle unfavorable views. We leverage priors learned from favorable images by feeding recentered images into a pretrained model augmented with low-rank adaptation (LoRA) layers. We further propose a Gaussian adapter module to enhance the geometric consistency of the Gaussians derived from the recentered inputs, along with a Gaussian alignment method to render accurate target views for training. Additionally, we introduce a new training strategy that utilizes an off-the-shelf dataset composed solely of favorable images. Experimental results on both synthetic images from the Google Scanned Objects dataset and real images from the OmniObject3D dataset validate the effectiveness of our method in handling unfavorable input views.