Borrowing from yourself: Faster future video segmentation with partial channel update

TL;DR

This paper introduces a novel method for future video segmentation that uses time-dependent channel masking to reduce computation and latency, leveraging previous frame features for faster, accurate predictions.

Contribution

It proposes a new channel masking technique for fast future video segmentation that improves efficiency by updating only a subset of feature maps at each step.

Findings

Reduced computation and latency in video segmentation models.

Effective use of previous frame features for future prediction.

Validated benefits across multiple architectures.

Abstract

Semantic segmentation is a well-addressed topic in the computer vision literature, but the design of fast and accurate video processing networks remains challenging. In addition, to run on embedded hardware, computer vision models often have to make compromises on accuracy to run at the required speed, so that a latency/accuracy trade-off is usually at the heart of these real-time systems' design. For the specific case of videos, models have the additional possibility to make use of computations made for previous frames to mitigate the accuracy loss while being real-time. In this work, we propose to tackle the task of fast future video segmentation prediction through the use of convolutional layers with time-dependent channel masking. This technique only updates a chosen subset of the feature maps at each time-step, bringing simultaneously less computation and latency, and allowing the network to leverage previously computed features. We apply this technique to several fast architectures and experimentally confirm its benefits for the future prediction subtask.