Online Model Distillation for Efficient Video Inference

TL;DR

This paper introduces an online model distillation approach to create efficient, specialized semantic segmentation models for live video streams, achieving high accuracy with significantly reduced inference costs without offline pretraining.

Contribution

It presents a novel online distillation method that trains low-cost models directly on live video, outperforming flow-based and video segmentation methods in efficiency and accuracy.

Findings

Achieves 7-17x lower runtime cost compared to teacher models.

Maintains high accuracy even on non-stationary video streams.

Provides a new dataset for evaluating long-term video inference efficiency.

Abstract

High-quality computer vision models typically address the problem of understanding the general distribution of real-world images. However, most cameras observe only a very small fraction of this distribution. This offers the possibility of achieving more efficient inference by specializing compact, low-cost models to the specific distribution of frames observed by a single camera. In this paper, we employ the technique of model distillation (supervising a low-cost student model using the output of a high-cost teacher) to specialize accurate, low-cost semantic segmentation models to a target video stream. Rather than learn a specialized student model on offline data from the video stream, we train the student in an online fashion on the live video, intermittently running the teacher to provide a target for learning. Online model distillation yields semantic segmentation models that closely approximate their Mask R-CNN teacher with 7 to 17$\times$ lower inference runtime cost (11 to 26$\times$ in FLOPs), even when the target video's distribution is non-stationary. Our method requires no offline pretraining on the target video stream, achieves higher accuracy and lower cost than solutions based on flow or video object segmentation, and can exhibit better temporal stability than the original teacher. We also provide a new video dataset for evaluating the efficiency of inference over long running video streams.