Video Semantic Segmentation with Distortion-Aware Feature Correction

TL;DR

This paper introduces a distortion-aware feature correction method for video semantic segmentation that effectively rectifies distorted features caused by optical flow inaccuracies, enhancing segmentation accuracy efficiently.

Contribution

The paper proposes a novel distortion map prediction and feature correction module to improve feature propagation in video segmentation, addressing flow inaccuracy issues.

Findings

Outperforms state-of-the-art methods on Cityscapes and CamVid datasets.

Significantly improves segmentation accuracy with low additional cost.

Effective correction of distorted propagated features enhances overall performance.

Abstract

Video semantic segmentation is active in recent years benefited from the great progress of image semantic segmentation. For such a task, the per-frame image segmentation is generally unacceptable in practice due to high computation cost. To tackle this issue, many works use the flow-based feature propagation to reuse the features of previous frames. However, the optical flow estimation inevitably suffers inaccuracy and then causes the propagated features distorted. In this paper, we propose distortion-aware feature correction to alleviate the issue, which improves video segmentation performance by correcting distorted propagated features. To be specific, we firstly propose to transfer distortion patterns from feature into image space and conduct effective distortion map prediction. Benefited from the guidance of distortion maps, we proposed Feature Correction Module (FCM) to rectify propagated features in the distorted areas. Our proposed method can significantly boost the accuracy of video semantic segmentation at a low price. The extensive experimental results on Cityscapes and CamVid show that our method outperforms the recent state-of-the-art methods.