ISDA: Position-Aware Instance Segmentation with Deformable Attention

TL;DR

ISDA introduces an end-to-end, NMS-free instance segmentation approach using deformable attention and position-aware kernels, achieving superior performance on MS-COCO compared to traditional methods.

Contribution

The paper presents a novel set-prediction based instance segmentation method that is end-to-end trainable and does not require NMS, leveraging deformable attention for improved accuracy.

Findings

Outperforms Mask R-CNN by 2.6 points on MS-COCO

NMS-free and end-to-end trainable architecture

Leverages deformable attention with multi-scale features

Abstract

Most instance segmentation models are not end-to-end trainable due to either the incorporation of proposal estimation (RPN) as a pre-processing or non-maximum suppression (NMS) as a post-processing. Here we propose a novel end-to-end instance segmentation method termed ISDA. It reshapes the task into predicting a set of object masks, which are generated via traditional convolution operation with learned position-aware kernels and features of objects. Such kernels and features are learned by leveraging a deformable attention network with multi-scale representation. Thanks to the introduced set-prediction mechanism, the proposed method is NMS-free. Empirically, ISDA outperforms Mask R-CNN (the strong baseline) by 2.6 points on MS-COCO, and achieves leading performance compared with recent models. Code will be available soon.