Masked Wavelet Representation for Compact Neural Radiance Fields

TL;DR

This paper introduces a wavelet-based compression method for neural radiance fields that significantly reduces memory usage while maintaining high-quality scene representation, enabling efficient rendering within a 2MB memory limit.

Contribution

The authors propose a novel wavelet transform and trainable masking approach to compress grid-based neural fields, achieving state-of-the-art performance with reduced memory requirements.

Findings

Higher sparsity in wavelet coefficients improves compression.

State-of-the-art results within 2MB memory budget.

Effective trade-off between compression and reconstruction quality.

Abstract

Neural radiance fields (NeRF) have demonstrated the potential of coordinate-based neural representation (neural fields or implicit neural representation) in neural rendering. However, using a multi-layer perceptron (MLP) to represent a 3D scene or object requires enormous computational resources and time. There have been recent studies on how to reduce these computational inefficiencies by using additional data structures, such as grids or trees. Despite the promising performance, the explicit data structure necessitates a substantial amount of memory. In this work, we present a method to reduce the size without compromising the advantages of having additional data structures. In detail, we propose using the wavelet transform on grid-based neural fields. Grid-based neural fields are for fast convergence, and the wavelet transform, whose efficiency has been demonstrated in high-performance standard codecs, is to improve the parameter efficiency of grids. Furthermore, in order to achieve a higher sparsity of grid coefficients while maintaining reconstruction quality, we present a novel trainable masking approach. Experimental results demonstrate that non-spatial grid coefficients, such as wavelet coefficients, are capable of attaining a higher level of sparsity than spatial grid coefficients, resulting in a more compact representation. With our proposed mask and compression pipeline, we achieved state-of-the-art performance within a memory budget of 2 MB. Our code is available at https://github.com/daniel03c1/masked_wavelet_nerf.