INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

TL;DR

This paper introduces INT, a method that improves task-generic promptable segmentation by adaptively mining negative examples to refine instance-specific prompts, enhancing segmentation accuracy across diverse datasets.

Contribution

INT presents a novel negative mining approach that adaptively filters irrelevant information to generate more accurate instance-specific prompts for segmentation.

Findings

Effective across six diverse datasets.

Improves segmentation robustness and scalability.

Outperforms existing prompt-based segmentation methods.

Abstract

Task-generic promptable image segmentation aims to achieve segmentation of diverse samples under a single task description by utilizing only one task-generic prompt. Current methods leverage the generalization capabilities of Vision-Language Models (VLMs) to infer instance-specific prompts from these task-generic prompts in order to guide the segmentation process. However, when VLMs struggle to generalise to some image instances, predicting instance-specific prompts becomes poor. To solve this problem, we introduce \textbf{I}nstance-specific \textbf{N}egative Mining for \textbf{T}ask-Generic Promptable Segmentation (\textbf{INT}). The key idea of INT is to adaptively reduce the influence of irrelevant (negative) prior knowledge whilst to increase the use the most plausible prior knowledge, selected by negative mining with higher contrast, in order to optimise instance-specific prompts generation. Specifically, INT consists of two components: (1) instance-specific prompt generation, which progressively fliters out incorrect information in prompt generation; (2) semantic mask generation, which ensures each image instance segmentation matches correctly the semantics of the instance-specific prompts. INT is validated on six datasets, including camouflaged objects and medical images, demonstrating its effectiveness, robustness and scalability.