AISFormer: Amodal Instance Segmentation with Transformer

TL;DR

AISFormer introduces a transformer-based framework for amodal instance segmentation that models complex mask coherence, outperforming previous CNN-based methods on multiple challenging benchmarks.

Contribution

The paper presents AISFormer, a novel transformer-based model that explicitly captures high-level feature coherence for amodal segmentation, improving over prior CNN-based approaches.

Findings

Achieves state-of-the-art results on KINS, D2SA, and COCOA-cls benchmarks.

Effectively models occluder, visible, amodal, and invisible masks with learnable queries.

Demonstrates significant improvements through extensive ablation studies.

Abstract

Amodal Instance Segmentation (AIS) aims to segment the region of both visible and possible occluded parts of an object instance. While Mask R-CNN-based AIS approaches have shown promising results, they are unable to model high-level features coherence due to the limited receptive field. The most recent transformer-based models show impressive performance on vision tasks, even better than Convolution Neural Networks (CNN). In this work, we present AISFormer, an AIS framework, with a Transformer-based mask head. AISFormer explicitly models the complex coherence between occluder, visible, amodal, and invisible masks within an object's regions of interest by treating them as learnable queries. Specifically, AISFormer contains four modules: (i) feature encoding: extract ROI and learn both short-range and long-range visual features. (ii) mask transformer decoding: generate the occluder, visible, and amodal mask query embeddings by a transformer decoder (iii) invisible mask embedding: model the coherence between the amodal and visible masks, and (iv) mask predicting: estimate output masks including occluder, visible, amodal and invisible. We conduct extensive experiments and ablation studies on three challenging benchmarks i.e. KINS, D2SA, and COCOA-cls to evaluate the effectiveness of AISFormer. The code is available at: https://github.com/UARK-AICV/AISFormer