Towards Scalable Neural Representation for Diverse Videos

TL;DR

This paper introduces D-NeRV, a neural representation framework that efficiently encodes diverse, long videos by decoupling content and motion, incorporating temporal reasoning, and using task-oriented flow, outperforming existing methods in compression and recognition.

Contribution

The paper presents D-NeRV, a novel neural video representation model that improves scalability and compression of diverse videos by decoupling content and motion, and integrating temporal reasoning.

Findings

D-NeRV surpasses NeRV and traditional compression in UCF101 and UVG datasets.

D-NeRV achieves 3%-10% higher accuracy in action recognition tasks.

Encoding long, diverse videos jointly improves compression results.

Abstract

Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images, and have been recently applied to encode videos (e.g., NeRV, E-NeRV). While achieving promising results, existing INR-based methods are limited to encoding a handful of short videos (e.g., seven 5-second videos in the UVG dataset) with redundant visual content, leading to a model design that fits individual video frames independently and is not efficiently scalable to a large number of diverse videos. This paper focuses on developing neural representations for a more practical setup -- encoding long and/or a large number of videos with diverse visual content. We first show that instead of dividing videos into small subsets and encoding them with separate models, encoding long and diverse videos jointly with a unified model achieves better compression results. Based on this observation, we propose D-NeRV, a novel neural representation framework designed to encode diverse videos by (i) decoupling clip-specific visual content from motion information, (ii) introducing temporal reasoning into the implicit neural network, and (iii) employing the task-oriented flow as intermediate output to reduce spatial redundancies. Our new model largely surpasses NeRV and traditional video compression techniques on UCF101 and UVG datasets on the video compression task. Moreover, when used as an efficient data-loader, D-NeRV achieves 3%-10% higher accuracy than NeRV on action recognition tasks on the UCF101 dataset under the same compression ratios.