Compressed Vision for Efficient Video Understanding

TL;DR

This paper introduces a neural compression-based framework for efficient processing of hour-long videos, enabling training and analysis on longer videos with standard hardware, and addressing augmentation challenges in compressed domain.

Contribution

The authors propose a novel neural compression approach that allows direct input of compressed videos into standard networks, significantly improving efficiency for long video processing.

Findings

Efficient training on Kinetics600 and COIN benchmarks.

Successful processing of hour-long videos using compressed representations.

Introduction of a latent code augmentation network.

Abstract

Experience and reasoning occur across multiple temporal scales: milliseconds, seconds, hours or days. The vast majority of computer vision research, however, still focuses on individual images or short videos lasting only a few seconds. This is because handling longer videos require more scalable approaches even to process them. In this work, we propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos. We replace standard video compression, e.g. JPEG, with neural compression and show that we can directly feed compressed videos as inputs to regular video networks. Operating on compressed videos improves efficiency at all pipeline levels -- data transfer, speed and memory -- making it possible to train models faster and on much longer videos. Processing compressed signals has, however, the downside of precluding standard augmentation techniques if done naively. We address that by introducing a small network that can apply transformations to latent codes corresponding to commonly used augmentations in the original video space. We demonstrate that with our compressed vision pipeline, we can train video models more efficiently on popular benchmarks such as Kinetics600 and COIN. We also perform proof-of-concept experiments with new tasks defined over hour-long videos at standard frame rates. Processing such long videos is impossible without using compressed representation.