HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

TL;DR

HyperReel introduces a novel 6-DoF video representation that combines a ray-conditioned sampling network with a compact volume model, enabling real-time, high-quality rendering of challenging scenes with minimal memory usage.

Contribution

The paper presents HyperReel, a new 6-DoF video method that achieves high-fidelity, real-time rendering with low memory footprint, surpassing prior approaches in quality and efficiency.

Findings

Achieves up to 18 fps at megapixel resolution.

Provides high visual quality with small memory footprint.

Operates without custom CUDA code.

Abstract

Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.