CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering

TL;DR

This paper introduces CLiFT, a neural rendering method using compressed light-field tokens that enables efficient, adaptable scene rendering with high quality and significant data reduction, suitable for various compute budgets.

Contribution

The paper presents a novel scene representation called CLiFT that allows for compute-efficient, adaptive neural rendering by compressing scene information into tokens, enabling flexible rendering quality and speed.

Findings

Achieves high-quality rendering with significantly reduced data size.

Provides a flexible trade-off between rendering quality and computational cost.

Outperforms existing methods on RealEstate10K and DL3DV datasets.

Abstract

This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene. CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network. Concretely, given a set of images, multi-view encoder tokenizes the images with the camera poses. Latent-space K-means selects a reduced set of rays as cluster centroids using the tokens. The multi-view ``condenser'' compresses the information of all the tokens into the centroid tokens to construct CLiFTs. At test time, given a target view and a compute budget (i.e., the number of CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute-adaptive renderer. Extensive experiments on RealEstate10K and DL3DV datasets quantitatively and qualitatively validate our approach, achieving significant data reduction with comparable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed.