TBN-ViT: Temporal Bilateral Network with Vision Transformer for Video Scene Parsing

TL;DR

This paper introduces TBN-ViT, a novel video scene parsing model combining convolutional spatial features, vision transformer context, and inter-frame temporal information to improve accuracy on the VSPW dataset.

Contribution

The paper proposes a Temporal Bilateral Network with Vision Transformer that effectively integrates spatial, contextual, and temporal features for enhanced video scene parsing.

Findings

Achieves 49.85% mIoU on VSPW2021 test dataset

Combines convolutional and transformer-based features effectively

Outperforms previous methods on the VSPW benchmark

Abstract

Video scene parsing in the wild with diverse scenarios is a challenging and great significance task, especially with the rapid development of automatic driving technique. The dataset Video Scene Parsing in the Wild(VSPW) contains well-trimmed long-temporal, dense annotation and high resolution clips. Based on VSPW, we design a Temporal Bilateral Network with Vision Transformer. We first design a spatial path with convolutions to generate low level features which can preserve the spatial information. Meanwhile, a context path with vision transformer is employed to obtain sufficient context information. Furthermore, a temporal context module is designed to harness the inter-frames contextual information. Finally, the proposed method can achieve the mean intersection over union(mIoU) of 49.85\% for the VSPW2021 Challenge test dataset.