Instances as Queries

TL;DR

QueryInst introduces a novel query-based instance segmentation method that leverages intrinsic one-to-one correspondence in object queries, achieving state-of-the-art results with improved speed on multiple benchmarks.

Contribution

The paper proposes QueryInst, a new query-based instance segmentation framework that simplifies multi-stage mask head connections and enhances performance and efficiency.

Findings

Outperforms HTC on COCO with 48.1 box AP and 42.8 mask AP

Achieves best online VIS performance with good speed-accuracy trade-off

Runs 2.4 times faster than comparable methods

Abstract

Recently, query based object detection frameworks achieve comparable performance with previous state-of-the-art object detectors. However, how to fully leverage such frameworks to perform instance segmentation remains an open problem. In this paper, we present QueryInst (Instances as Queries), a query based instance segmentation method driven by parallel supervision on dynamic mask heads. The key insight of QueryInst is to leverage the intrinsic one-to-one correspondence in object queries across different stages, as well as one-to-one correspondence between mask RoI features and object queries in the same stage. This approach eliminates the explicit multi-stage mask head connection and the proposal distribution inconsistency issues inherent in non-query based multi-stage instance segmentation methods. We conduct extensive experiments on three challenging benchmarks, i.e., COCO, CityScapes, and YouTube-VIS to evaluate the effectiveness of QueryInst in instance segmentation and video instance segmentation (VIS) task. Specifically, using ResNet-101-FPN backbone, QueryInst obtains 48.1 box AP and 42.8 mask AP on COCO test-dev, which is 2 points higher than HTC in terms of both box AP and mask AP, while runs 2.4 times faster. For video instance segmentation, QueryInst achieves the best performance among all online VIS approaches and strikes a decent speed-accuracy trade-off. Code is available at \url{https://github.com/hustvl/QueryInst}.