Deep Video Codec Control for Vision Models

TL;DR

This paper introduces a novel end-to-end learnable deep video codec control that optimizes both bandwidth usage and downstream vision model performance, outperforming traditional standard codecs in preserving vision tasks.

Contribution

It presents the first approach to integrate deep vision performance considerations into standard-compliant video codec control.

Findings

Deep vision models' performance deteriorates with standard video codecs.

The proposed method better preserves downstream vision performance.

Our approach outperforms traditional codecs in vision tasks.

Abstract

Standardized lossy video coding is at the core of almost all real-world video processing pipelines. Rate control is used to enable standard codecs to adapt to different network bandwidth conditions or storage constraints. However, standard video codecs (e.g., H.264) and their rate control modules aim to minimize video distortion w.r.t. human quality assessment. We demonstrate empirically that standard-coded videos vastly deteriorate the performance of deep vision models. To overcome the deterioration of vision performance, this paper presents the first end-to-end learnable deep video codec control that considers both bandwidth constraints and downstream deep vision performance, while adhering to existing standardization. We demonstrate that our approach better preserves downstream deep vision performance than traditional standard video coding.