HAFormer: Unleashing the Power of Hierarchy-Aware Features for Lightweight Semantic Segmentation

TL;DR

HAFormer is a lightweight semantic segmentation model that effectively combines hierarchical CNN features with Transformer-based global context modeling, achieving high accuracy and speed with minimal computational cost.

Contribution

The paper introduces HAFormer, integrating a Hierarchy-Aware Pixel-Excitation module, an Efficient Transformer, and a correlation-weighted Fusion module for improved lightweight segmentation.

Findings

Achieves 74.2% mIoU on Cityscapes with high frame rate

Outperforms existing lightweight models in accuracy and efficiency

Maintains low computational overhead with compact model size

Abstract

Both Convolutional Neural Networks (CNNs) and Transformers have shown great success in semantic segmentation tasks. Efforts have been made to integrate CNNs with Transformer models to capture both local and global context interactions. However, there is still room for enhancement, particularly when considering constraints on computational resources. In this paper, we introduce HAFormer, a model that combines the hierarchical features extraction ability of CNNs with the global dependency modeling capability of Transformers to tackle lightweight semantic segmentation challenges. Specifically, we design a Hierarchy-Aware Pixel-Excitation (HAPE) module for adaptive multi-scale local feature extraction. During the global perception modeling, we devise an Efficient Transformer (ET) module streamlining the quadratic calculations associated with traditional Transformers. Moreover, a correlation-weighted Fusion (cwF) module selectively merges diverse feature representations, significantly enhancing predictive accuracy. HAFormer achieves high performance with minimal computational overhead and compact model size, achieving 74.2% mIoU on Cityscapes and 71.1% mIoU on CamVid test datasets, with frame rates of 105FPS and 118FPS on a single 2080Ti GPU. The source codes are available at https://github.com/XU-GITHUB-curry/HAFormer.