Scalable Video Coding for Humans and Machines

TL;DR

This paper introduces a scalable video coding framework that efficiently supports both machine vision and human viewing, achieving significant bit savings for object detection while maintaining visual quality.

Contribution

It presents a novel scalable coding framework combining conventional and DNN-based methods to optimize video for both humans and machines.

Findings

13-19% bit savings for object detection

Competitive MS-SSIM for human vision

Effective dual-purpose video coding

Abstract

Video content is watched not only by humans, but increasingly also by machines. For example, machine learning models analyze surveillance video for security and traffic monitoring, search through YouTube videos for inappropriate content, and so on. In this paper, we propose a scalable video coding framework that supports machine vision (specifically, object detection) through its base layer bitstream and human vision via its enhancement layer bitstream. The proposed framework includes components from both conventional and Deep Neural Network (DNN)-based video coding. The results show that on object detection, the proposed framework achieves 13-19% bit savings compared to state-of-the-art video codecs, while remaining competitive in terms of MS-SSIM on the human vision task.