Fully Transformer Networks for Semantic Image Segmentation

TL;DR

This paper introduces a novel Fully Transformer Network framework for semantic image segmentation, utilizing a Pyramid Group Transformer encoder and a Feature Pyramid Transformer for multi-level feature fusion, achieving superior results on multiple benchmarks.

Contribution

The paper proposes a pure Transformer-based encoder-decoder framework for semantic segmentation, combining hierarchical feature learning and multi-level feature fusion, outperforming existing methods.

Findings

Achieved better results on PASCAL Context, ADE20K, COCOStuff, and CelebAMask-HQ.

Introduced Pyramid Group Transformer (PGT) for efficient hierarchical feature learning.

Developed Feature Pyramid Transformer (FPT) for effective multi-level feature fusion.

Abstract

Transformers have shown impressive performance in various natural language processing and computer vision tasks, due to the capability of modeling long-range dependencies. Recent progress has demonstrated that combining such Transformers with CNN-based semantic image segmentation models is very promising. However, it is not well studied yet on how well a pure Transformer based approach can achieve for image segmentation. In this work, we explore a novel framework for semantic image segmentation, which is encoder-decoder based Fully Transformer Networks (FTN). Specifically, we first propose a Pyramid Group Transformer (PGT) as the encoder for progressively learning hierarchical features, meanwhile reducing the computation complexity of the standard Visual Transformer (ViT). Then, we propose a Feature Pyramid Transformer (FPT) to fuse semantic-level and spatial-level information from multiple levels of the PGT encoder for semantic image segmentation. Surprisingly, this simple baseline can achieve better results on multiple challenging semantic segmentation and face parsing benchmarks, including PASCAL Context, ADE20K, COCOStuff, and CelebAMask-HQ. The source code will be released on https://github.com/BR-IDL/PaddleViT.