WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild

TL;DR

WildCAT3D introduces a novel scene-level view synthesis method that leverages diverse, in-the-wild data by modeling global appearance variations, enabling consistent multi-view generation and appearance control.

Contribution

It extends multi-view diffusion models to learn from diverse, uncurated scene images with appearance variations, improving scene-level NVS with less data.

Findings

State-of-the-art single-view NVS results.

Effective training on less diverse data sources.

Enables global appearance control during generation.

Abstract

Despite recent advances in sparse novel view synthesis (NVS) applied to object-centric scenes, scene-level NVS remains a challenge. A central issue is the lack of available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and permissively-licensed data exists in the wild, consisting of scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such as tourist photos. To this end, we present WildCAT3D, a framework for generating novel views of scenes learned from diverse 2D scene image data captured in the wild. We unlock training on these data sources by explicitly modeling global appearance conditions in images, extending the state-of-the-art multi-view diffusion paradigm to learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-art results on single-view NVS in object- and scene-level settings, while training on strictly less data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.