Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation

TL;DR

This paper introduces a novel distortion-aware network pruning framework that leverages temporal locality in videos to accelerate real-time semantic segmentation with minimal accuracy loss.

Contribution

It proposes a dynamic feature reuse and partial computation method using a spatial-temporal mask generator to improve speed in real-time video segmentation.

Findings

Significant speed-up in inference time across multiple benchmarks.

Minimal accuracy degradation with the proposed method.

Effective dynamic block dropping based on inter-frame distortion.

Abstract

Real-time video segmentation is a crucial task for many real-world applications such as autonomous driving and robot control. Since state-of-the-art semantic segmentation models are often too heavy for real-time applications despite their impressive performance, researchers have proposed lightweight architectures with speed-accuracy trade-offs, achieving real-time speed at the expense of reduced accuracy. In this paper, we propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks by exploiting the temporal locality in videos. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame. This is done by dynamically dropping out residual blocks using a gating mechanism which decides which blocks to drop based on inter-frame distortion. We validate our Spatial-Temporal Mask Generator (STMG) on video semantic segmentation benchmarks with multiple backbone networks, and show that our method largely speeds up inference with minimal loss of accuracy.