Lightplane: Highly-Scalable Components for Neural 3D Fields

TL;DR

Lightplane introduces scalable components that drastically reduce memory requirements for neural 3D fields, enabling high-resolution 3D reconstruction and generation from large image datasets.

Contribution

The paper presents Lightplane Render and Splatter, novel scalable modules that improve memory efficiency in 2D-3D mapping for neural fields.

Findings

Enables processing of more high-resolution images with less memory.

Improves scalability of 3D reconstruction and generation pipelines.

Demonstrates utility across various 3D applications.

Abstract

Contemporary 3D research, particularly in reconstruction and generation, heavily relies on 2D images for inputs or supervision. However, current designs for these 2D-3D mapping are memory-intensive, posing a significant bottleneck for existing methods and hindering new applications. In response, we propose a pair of highly scalable components for 3D neural fields: Lightplane Render and Splatter, which significantly reduce memory usage in 2D-3D mapping. These innovations enable the processing of vastly more and higher resolution images with small memory and computational costs. We demonstrate their utility in various applications, from benefiting single-scene optimization with image-level losses to realizing a versatile pipeline for dramatically scaling 3D reconstruction and generation. Code: \url{https://github.com/facebookresearch/lightplane}.