VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations

TL;DR

VQ-NeRF introduces a vector quantization-based method that reduces computational complexity in neural radiance fields, enabling faster rendering while maintaining high-quality 3D reconstructions through multi-scale sampling and semantic loss.

Contribution

The paper presents a novel pipeline combining vector quantization, multi-scale sampling, and semantic loss to improve efficiency and detail preservation in implicit neural representations.

Findings

Achieves a better balance between rendering quality and efficiency.

Outperforms existing methods on multiple datasets.

Effectively preserves fine scene details despite high compression.

Abstract

Recent advancements in implicit neural representations have contributed to high-fidelity surface reconstruction and photorealistic novel view synthesis. However, the computational complexity inherent in these methodologies presents a substantial impediment, constraining the attainable frame rates and resolutions in practical applications. In response to this predicament, we propose VQ-NeRF, an effective and efficient pipeline for enhancing implicit neural representations via vector quantization. The essence of our method involves reducing the sampling space of NeRF to a lower resolution and subsequently reinstating it to the original size utilizing a pre-trained VAE decoder, thereby effectively mitigating the sampling time bottleneck encountered during rendering. Although the codebook furnishes representative features, reconstructing fine texture details of the scene remains challenging due to high compression rates. To overcome this constraint, we design an innovative multi-scale NeRF sampling scheme that concurrently optimizes the NeRF model at both compressed and original scales to enhance the network's ability to preserve fine details. Furthermore, we incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions. Extensive experiments demonstrate the effectiveness of our model in achieving the optimal trade-off between rendering quality and efficiency. Evaluation on the DTU, BlendMVS, and H3DS datasets confirms the superior performance of our approach.