Delta Distillation for Efficient Video Processing

TL;DR

This paper introduces Delta Distillation, a novel knowledge distillation method that leverages temporal redundancies in videos to improve the efficiency of video processing tasks like object detection and segmentation.

Contribution

The paper proposes Delta Distillation, a new approach that distills temporal variations in features, enabling more efficient video processing without relying on motion alignment techniques.

Findings

Sets new state-of-the-art accuracy-efficiency trade-offs.

Improves temporal consistency of teacher models.

Effective across various architectures, including efficient ones.

Abstract

This paper aims to accelerate video stream processing, such as object detection and semantic segmentation, by leveraging the temporal redundancies that exist between video frames. Instead of propagating and warping features using motion alignment, such as optical flow, we propose a novel knowledge distillation schema coined as Delta Distillation. In our proposal, the student learns the variations in the teacher's intermediate features over time. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. During inference, both teacher and student cooperate for providing predictions: the former by providing initial representations extracted only on the key-frame, and the latter by iteratively estimating and applying deltas for the successive frames. Moreover, we consider various design choices to learn optimal student architectures including an end-to-end learnable architecture search. By extensive experiments on a wide range of architectures, including the most efficient ones, we demonstrate that delta distillation sets a new state of the art in terms of accuracy vs. efficiency trade-off for semantic segmentation and object detection in videos. Finally, we show that, as a by-product, delta distillation improves the temporal consistency of the teacher model.