DORSal: Diffusion for Object-centric Representations of Scenes et al

TL;DR

DORSal introduces a diffusion-based approach for high-fidelity 3D scene rendering that maintains object-level editing capabilities, outperforming existing methods on synthetic and real-world datasets.

Contribution

This paper adapts a diffusion architecture for 3D scene generation conditioned on object-centric representations, enabling scalable, high-quality rendering with editing features.

Findings

Achieves high-quality novel view synthesis in complex scenes

Enables object-level scene editing in 3D representations

Outperforms existing approaches on synthetic and real-world datasets

Abstract

Recent progress in 3D scene understanding enables scalable learning of representations across large datasets of diverse scenes. As a consequence, generalization to unseen scenes and objects, rendering novel views from just a single or a handful of input images, and controllable scene generation that supports editing, is now possible. However, training jointly on a large number of scenes typically compromises rendering quality when compared to single-scene optimized models such as NeRFs. In this paper, we leverage recent progress in diffusion models to equip 3D scene representation learning models with the ability to render high-fidelity novel views, while retaining benefits such as object-level scene editing to a large degree. In particular, we propose DORSal, which adapts a video diffusion architecture for 3D scene generation conditioned on frozen object-centric slot-based representations of scenes. On both complex synthetic multi-object scenes and on the real-world large-scale Street View dataset, we show that DORSal enables scalable neural rendering of 3D scenes with object-level editing and improves upon existing approaches.