CCNet: Criss-Cross Attention for Semantic Segmentation

TL;DR

CCNet introduces a novel criss-cross attention module that efficiently captures full-image contextual information for semantic segmentation, achieving state-of-the-art results with reduced memory and computational costs.

Contribution

The paper proposes a recurrent criss-cross attention module that is GPU memory friendly, computationally efficient, and effective for full-image context modeling in semantic segmentation.

Findings

Achieves state-of-the-art mIoU scores on Cityscapes, ADE20K, and LIP benchmarks.

Requires 11x less GPU memory than non-local blocks.

Reduces FLOPs by approximately 85% compared to non-local attention.

Abstract

Contextual information is vital in visual understanding problems, such as semantic segmentation and object detection. We propose a Criss-Cross Network (CCNet) for obtaining full-image contextual information in a very effective and efficient way. Concretely, for each pixel, a novel criss-cross attention module harvests the contextual information of all the pixels on its criss-cross path. By taking a further recurrent operation, each pixel can finally capture the full-image dependencies. Besides, a category consistent loss is proposed to enforce the criss-cross attention module to produce more discriminative features. Overall, CCNet is with the following merits: 1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross attention module requires 11x less GPU memory usage. 2) High computational efficiency. The recurrent criss-cross attention significantly reduces FLOPs by about 85% of the non-local block. 3) The state-of-the-art performance. We conduct extensive experiments on semantic segmentation benchmarks including Cityscapes, ADE20K, human parsing benchmark LIP, instance segmentation benchmark COCO, video segmentation benchmark CamVid. In particular, our CCNet achieves the mIoU scores of 81.9%, 45.76% and 55.47% on the Cityscapes test set, the ADE20K validation set and the LIP validation set respectively, which are the new state-of-the-art results. The source codes are available at \url{https://github.com/speedinghzl/CCNet}.