JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression

TL;DR

JointRF introduces an end-to-end optimized method for representing and compressing dynamic Neural Radiance Fields, significantly improving quality and efficiency for volumetric video rendering.

Contribution

It proposes a novel joint optimization scheme combining dynamic NeRF representation with compression, utilizing residual and coefficient feature grids for better handling of large motions and redundancy.

Findings

Achieves superior compression performance on multiple datasets.

Handles large motions without quality loss.

Reduces temporal and spatial redundancy effectively.

Abstract

Neural Radiance Field (NeRF) excels in photo-realistically static scenes, inspiring numerous efforts to facilitate volumetric videos. However, rendering dynamic and long-sequence radiance fields remains challenging due to the significant data required to represent volumetric videos. In this paper, we propose a novel end-to-end joint optimization scheme of dynamic NeRF representation and compression, called JointRF, thus achieving significantly improved quality and compression efficiency against the previous methods. Specifically, JointRF employs a compact residual feature grid and a coefficient feature grid to represent the dynamic NeRF. This representation handles large motions without compromising quality while concurrently diminishing temporal redundancy. We also introduce a sequential feature compression subnetwork to further reduce spatial-temporal redundancy. Finally, the representation and compression subnetworks are end-to-end trained combined within the JointRF. Extensive experiments demonstrate that JointRF can achieve superior compression performance across various datasets.