Accelerating Video Object Segmentation with Compressed Video

TL;DR

This paper introduces a fast, plug-and-play framework for semi-supervised video object segmentation that leverages compressed video data to significantly speed up processing while maintaining high accuracy.

Contribution

It presents a novel motion vector-based warping and residual correction method that accelerates existing segmentation algorithms using compressed video features.

Findings

Achieved up to 3.5x speed-up on DAVIS17 and YouTube-VOS datasets.

Maintained competitive segmentation accuracy with minor drops.

Flexible approach compatible with multiple existing algorithms.

Abstract

We propose an efficient plug-and-play acceleration framework for semi-supervised video object segmentation by exploiting the temporal redundancies in videos presented by the compressed bitstream. Specifically, we propose a motion vector-based warping method for propagating segmentation masks from keyframes to other frames in a bi-directional and multi-hop manner. Additionally, we introduce a residual-based correction module that can fix wrongly propagated segmentation masks from noisy or erroneous motion vectors. Our approach is flexible and can be added on top of several existing video object segmentation algorithms. We achieved highly competitive results on DAVIS17 and YouTube-VOS on various base models with substantial speed-ups of up to 3.5X with minor drops in accuracy.