Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time

TL;DR

This paper introduces a neural video decomposition approach that enables layer-based editing with consistent effects across frames, using a novel multiplicative residual model and efficient learning techniques for real-time performance.

Contribution

It proposes a new neural decomposition method with multiplicative residuals for spatiotemporal variations, enabling real-time, high-quality video editing with consistent effects.

Findings

Efficient learning of 1080p video decomposition in 25 seconds per frame.

Real-time rendering at 71 fps on a single GPU.

Effective high-quality editing demonstrated on various videos.

Abstract

We present a video decomposition method that facilitates layer-based editing of videos with spatiotemporally varying lighting and motion effects. Our neural model decomposes an input video into multiple layered representations, each comprising a 2D texture map, a mask for the original video, and a multiplicative residual characterizing the spatiotemporal variations in lighting conditions. A single edit on the texture maps can be propagated to the corresponding locations in the entire video frames while preserving other contents' consistencies. Our method efficiently learns the layer-based neural representations of a 1080p video in 25s per frame via coordinate hashing and allows real-time rendering of the edited result at 71 fps on a single GPU. Qualitatively, we run our method on various videos to show its effectiveness in generating high-quality editing effects. Quantitatively, we propose to adopt feature-tracking evaluation metrics for objectively assessing the consistency of video editing. Project page: https://lightbulb12294.github.io/hashing-nvd/