INV: Towards Streaming Incremental Neural Videos

TL;DR

This paper introduces Incremental Neural Videos (INV), a novel per-frame NeRF approach that enables efficient, lag-free streaming of photorealistic videos by leveraging the natural structure of MLPs to amortize training across frames.

Contribution

The paper presents a new incremental NeRF method that reduces training time and size, enabling interactive streaming without buffer lag by exploiting MLPs' structural properties.

Findings

INV achieves >28.6db quality in 8 minutes per frame.

Outperforms prior state-of-the-art methods with 19% less training time.

Reduces per-frame size to 0.3MB, suitable for streaming.

Abstract

Recent works in spatiotemporal radiance fields can produce photorealistic free-viewpoint videos. However, they are inherently unsuitable for interactive streaming scenarios (e.g. video conferencing, telepresence) because have an inevitable lag even if the training is instantaneous. This is because these approaches consume videos and thus have to buffer chunks of frames (often seconds) before processing. In this work, we take a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Conventional wisdom believes that per-frame NeRFs are impractical due to prohibitive training costs and storage. We break this belief by introducing Incremental Neural Videos (INV), a per-frame NeRF that is efficiently trained and streamable. We designed INV based on two insights: (1) Our main finding is that MLPs naturally partition themselves into Structure and Color Layers, which store structural and color/texture information respectively. (2) We leverage this property to retain and improve upon knowledge from previous frames, thus amortizing training across frames and reducing redundant learning. As a result, with negligible changes to NeRF, INV can achieve good qualities (>28.6db) in 8min/frame. It can also outperform prior SOTA in 19% less training time. Additionally, our Temporal Weight Compression reduces the per-frame size to 0.3MB/frame (6.6% of NeRF). More importantly, INV is free from buffer lag and is naturally fit for streaming. While this work does not achieve real-time training, it shows that incremental approaches like INV present new possibilities in interactive 3D streaming. Moreover, our discovery of natural information partition leads to a better understanding and manipulation of MLPs. Code and dataset will be released soon.