CLUSTSEG: Clustering for Universal Segmentation

TL;DR

CLUSTSEG introduces a unified transformer-based clustering framework capable of handling various image segmentation tasks by flexible cluster initialization and an EM-like iterative process, achieving superior results without task-specific architecture changes.

Contribution

It proposes a novel, task-agnostic clustering scheme within a transformer framework, unifying multiple segmentation tasks with innovative initialization and assignment strategies.

Findings

Achieves state-of-the-art results across multiple segmentation tasks.

Flexible cluster initialization tailored to task demands.

No additional parameters needed for pixel-cluster assignment.

Abstract

We present CLUSTSEG, a general, transformer-based framework that tackles different image segmentation tasks (i.e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme. Regarding queries as cluster centers, CLUSTSEG is innovative in two aspects:1) cluster centers are initialized in heterogeneous ways so as to pointedly address task-specific demands (e.g., instance- or category-level distinctiveness), yet without modifying the architecture; and 2) pixel-cluster assignment, formalized in a cross-attention fashion, is alternated with cluster center update, yet without learning additional parameters. These innovations closely link CLUSTSEG to EM clustering and make it a transparent and powerful framework that yields superior results across the above segmentation tasks.