TensoRF: Tensorial Radiance Fields

TL;DR

TensoRF introduces a tensor-based scene representation for radiance fields that significantly improves rendering quality, reduces memory footprint, and accelerates reconstruction compared to traditional NeRF models.

Contribution

The paper proposes a novel tensor factorization framework for radiance fields, including CP and VM decompositions, leading to faster, more efficient, and higher-quality scene reconstructions.

Findings

TensoRF with CP decomposition achieves reconstruction in under 30 minutes with high quality.

VM decomposition further improves rendering quality and reduces reconstruction time to under 10 minutes.

Models are significantly smaller, with sizes under 4 MB for CP and 75 MB for VM, outperforming previous methods.

Abstract

We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).