Laser: Latent Set Representations for 3D Generative Modeling

TL;DR

Laser-NV is a novel 3D generative model that uses set-valued latent representations and geometry-informed attention to produce high-fidelity, diverse scene completions from limited views, advancing the capabilities of scene synthesis.

Contribution

It introduces Laser-NV, a set-based latent model with normalizing flows and attention mechanisms, enabling high-quality, fast, and diverse 3D scene synthesis from few views.

Findings

Achieves state-of-the-art view synthesis on ShapeNet and City datasets.

Produces diverse, plausible scene completions consistent with observed views.

Operates efficiently with fast feed-forward inference from limited input views.

Abstract

NeRF provides unparalleled fidelity of novel view synthesis: rendering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover a scene, which limits its applicability. While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts. We introduce Laser-NV: a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows. Similarly to previous amortized approaches, Laser-NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views. To encourage higher rendering fidelity and consistency with observed views, Laser-NV further incorporates a geometry-informed attention mechanism over the observed views. Laser-NV further produces diverse and plausible completions of occluded parts of a scene while remaining consistent with observations. Laser-NV shows state-of-the-art novel-view synthesis quality when evaluated on ShapeNet and on a novel simulated City dataset, which features high uncertainty in the unobserved regions of the scene.