Dynamic Focus-aware Positional Queries for Semantic Segmentation

TL;DR

This paper introduces Dynamic Focus-aware Positional Queries (DFPQ), a novel query design for semantic segmentation that dynamically generates positional queries based on cross-attention scores, leading to more accurate localization and state-of-the-art results.

Contribution

The paper proposes DFPQ, a new query design that enhances localization accuracy by dynamically generating positional queries conditioned on cross-attention scores, improving semantic segmentation performance.

Findings

Achieves state-of-the-art performance on ADE20K and Cityscapes datasets.

Outperforms Mask2former by 1.1% to 1.9% mIoU with various backbones.

Effectively handles high-resolution cross-attention with local relation aggregation.

Abstract

The DETR-like segmentors have underpinned the most recent breakthroughs in semantic segmentation, which end-to-end train a set of queries representing the class prototypes or target segments. Recently, masked attention is proposed to restrict each query to only attend to the foreground regions predicted by the preceding decoder block for easier optimization. Although promising, it relies on the learnable parameterized positional queries which tend to encode the dataset statistics, leading to inaccurate localization for distinct individual queries. In this paper, we propose a simple yet effective query design for semantic segmentation termed Dynamic Focus-aware Positional Queries (DFPQ), which dynamically generates positional queries conditioned on the cross-attention scores from the preceding decoder block and the positional encodings for the corresponding image features, simultaneously. Therefore, our DFPQ preserves rich localization information for the target segments and provides accurate and fine-grained positional priors. In addition, we propose to efficiently deal with high-resolution cross-attention by only aggregating the contextual tokens based on the low-resolution cross-attention scores to perform local relation aggregation. Extensive experiments on ADE20K and Cityscapes show that with the two modifications on Mask2former, our framework achieves SOTA performance and outperforms Mask2former by clear margins of 1.1%, 1.9%, and 1.1% single-scale mIoU with ResNet-50, Swin-T, and Swin-B backbones on the ADE20K validation set, respectively. Source code is available at https://github.com/ziplab/FASeg