CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen

TL;DR

This paper introduces CSL, a flexible framework that enhances segmentation models' ability to identify unseen and out-of-distribution classes by embedding structural constraints, improving performance across multiple challenging segmentation tasks.

Contribution

The paper presents CSL, a novel plug-in framework that integrates structural constraints into existing segmentation methods, improving unseen class segmentation without retraining.

Findings

CSL improves OOD segmentation performance.

CSL enhances zero-shot semantic segmentation accuracy.

CSL outperforms state-of-the-art methods across tasks.

Abstract

Addressing Out-Of-Distribution (OOD) Segmentation and Zero-Shot Semantic Segmentation (ZS3) is challenging, necessitating segmenting unseen classes. Existing strategies adapt the class-agnostic Mask2Former (CA-M2F) tailored to specific tasks. However, these methods cater to singular tasks, demand training from scratch, and we demonstrate certain deficiencies in CA-M2F, which affect performance. We propose the Class-Agnostic Structure-Constrained Learning (CSL), a plug-in framework that can integrate with existing methods, thereby embedding structural constraints and achieving performance gain, including the unseen, specifically OOD, ZS3, and domain adaptation (DA) tasks. There are two schemes for CSL to integrate with existing methods (1) by distilling knowledge from a base teacher network, enforcing constraints across training and inference phrases, or (2) by leveraging established models to obtain per-pixel distributions without retraining, appending constraints during the inference phase. We propose soft assignment and mask split methodologies that enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation, consistently transcending the state-of-art across all three tasks.