Efficient Video Segmentation Models with Per-frame Inference

TL;DR

This paper introduces a training-based approach to improve temporal consistency in video segmentation models without increasing inference computation, using constraints, loss functions, and knowledge distillation techniques.

Contribution

It proposes novel training methods including a temporal consistency loss and knowledge distillation to enhance video segmentation consistency during inference.

Findings

Outperforms keyframe-based and baseline methods on Cityscapes, Camvid, and 300VW-Mask datasets.

Achieves better temporal smoothness and accuracy without extra inference overhead.

Demonstrates applicability to video instance segmentation and portrait matting.

Abstract

Most existing real-time deep models trained with each frame independently may produce inconsistent results across the temporal axis when tested on a video sequence. A few methods take the correlations in the video sequence into account,e.g., by propagating the results to the neighboring frames using optical flow or extracting frame representations using multi-frame information, which may lead to inaccurate results or unbalanced latency. In this work, we focus on improving the temporal consistency without introducing computation overhead in inference. To this end, we perform inference at each frame. Temporal consistency is achieved by learning from video frames with extra constraints during the training phase. introduced for inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods. On the task of semantic video segmentation, weighing among accuracy, temporal smoothness, and efficiency, our proposed method outperforms keyframe-based methods and a few baseline methods that are trained with each frame independently, on datasets including Cityscapes, Camvid, and 300VW-Mask. We further apply our training method to video instance segmentation on YouTubeVISand develop an application of portrait matting in video sequences, by segmenting temporally consistent instance-level trimaps across frames. Experiments show superior qualitative and quantitative results. Code is available at: https://git.io/vidseg.