Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving

TL;DR

This paper introduces a Spatio-Temporal Attention mechanism that enhances transformer-based video semantic segmentation by leveraging multi-frame context, significantly improving temporal consistency and accuracy in dynamic driving scenes.

Contribution

It proposes a novel Spatio-Temporal Attention module that extends transformers to incorporate multi-frame information with minimal architectural changes.

Findings

9.20 percentage points improvement in temporal consistency

Up to 1.76 percentage points increase in mean IoU

Effective across various transformer architectures

Abstract

Deep neural networks, especially transformer-based architectures, have achieved remarkable success in semantic segmentation for environmental perception. However, existing models process video frames independently, thus failing to leverage temporal consistency, which could significantly improve both accuracy and stability in dynamic scenes. In this work, we propose a Spatio-Temporal Attention (STA) mechanism that extends transformer attention blocks to incorporate multi-frame context, enabling robust temporal feature representations for video semantic segmentation. Our approach modifies standard self-attention to process spatio-temporal feature sequences while maintaining computational efficiency and requiring minimal changes to existing architectures. STA demonstrates broad applicability across diverse transformer architectures and remains effective across both lightweight and larger-scale models. A comprehensive evaluation on the Cityscapes and BDD100k datasets shows substantial improvements of 9.20 percentage points in temporal consistency metrics and up to 1.76 percentage points in mean intersection over union compared to single-frame baselines. These results demonstrate STA as an effective architectural enhancement for video-based semantic segmentation applications.