Towards Instance Segmentation with Polygon Detection Transformers

TL;DR

Poly-DETR introduces a novel polygon detection transformer for instance segmentation that improves accuracy and efficiency by reformulating the task as sparse vertex regression, outperforming existing methods especially in high-resolution and domain-specific scenarios.

Contribution

The paper proposes Poly-DETR, a transformer-based approach that reformulates instance segmentation as polygon vertex regression, with novel attention and training schemes, achieving superior performance and efficiency.

Findings

Achieves 4.7 mAP improvement on MS COCO test-dev.

Reduces memory consumption by nearly half on Cityscapes.

Outperforms mask-based methods on domain-specific datasets.

Abstract

One of the bottlenecks for instance segmentation today lies in the conflicting requirements of high-resolution inputs and lightweight, real-time inference. To address this bottleneck, we present a Polygon Detection Transformer (Poly-DETR) to reformulate instance segmentation as sparse vertex regression via Polar Representation, thereby eliminating the reliance on dense pixel-wise mask prediction. Considering the box-to-polygon reference shift in Detection Transformers, we propose Polar Deformable Attention and Position-Aware Training Scheme to dynamically update supervision and focus attention on boundary cues. Compared with state-of-the-art polar-based methods, Poly-DETR achieves a 4.7 mAP improvement on MS COCO test-dev. Moreover, we construct a parallel mask-based counterpart to support a systematic comparison between polar and mask representations. Experimental results show that Poly-DETR is more lightweight in high-resolution scenarios, reducing memory consumption by almost half on Cityscapes dataset. Notably, on PanNuke (cell segmentation) and SpaceNet (building footprints) datasets, Poly-DETR surpasses its mask-based counterpart on all metrics, which validates its advantage on regular-shaped instances in domain-specific settings.