Local Memory Attention for Fast Video Semantic Segmentation

TL;DR

This paper introduces a fast, general local memory attention module that enhances video semantic segmentation by efficiently integrating past frame information, improving accuracy with minimal inference time increase.

Contribution

A novel local attention-based memory module that can be integrated into existing segmentation models to improve video segmentation performance efficiently.

Findings

Improved mIoU by 1.7% on Cityscapes with ERFNet

Enhanced mIoU by 2.1% on Cityscapes with PSPNet

Increased inference time by only 1.5ms for ERFNet

Abstract

We propose a novel neural network module that transforms an existing single-frame semantic segmentation model into a video semantic segmentation pipeline. In contrast to prior works, we strive towards a simple, fast, and general module that can be integrated into virtually any single-frame architecture. Our approach aggregates a rich representation of the semantic information in past frames into a memory module. Information stored in the memory is then accessed through an attention mechanism. In contrast to previous memory-based approaches, we propose a fast local attention layer, providing temporal appearance cues in the local region of prior frames. We further fuse these cues with an encoding of the current frame through a second attention-based module. The segmentation decoder processes the fused representation to predict the final semantic segmentation. We integrate our approach into two popular semantic segmentation networks: ERFNet and PSPNet. We observe an improvement in segmentation performance on Cityscapes by 1.7% and 2.1% in mIoU respectively, while increasing inference time of ERFNet by only 1.5ms.