Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive

TL;DR

Inter2Former introduces a dynamic hybrid attention framework that optimizes dense-token processing for interactive segmentation, achieving high accuracy and efficiency on CPU devices by adaptively focusing computation on regions of interest.

Contribution

The paper presents a novel adaptive computation framework with Dynamic Prompt Embedding, Hybrid Attention, Mixture of Experts, and Local Upsampling for efficient high-precision interactive segmentation.

Findings

Achieves state-of-the-art performance on high-precision IS benchmarks.

Operates efficiently on CPU devices with high segmentation quality.

Utilizes adaptive attention and computation strategies to focus on relevant regions.

Abstract

Interactive segmentation (IS) improves annotation efficiency by segmenting target regions from user prompts, with widespread applications in real-world scenarios. Current approaches face a critical trade-off: dense-token methods achieve superior accuracy and detail preservation but suffer from prohibitively slow processing on CPU devices, while the Segment Anything Model (SAM) advances the field with sparse prompt tokens for fast inference but compromises segmentation quality. In this paper, we propose Inter2Former to address this challenge by optimizing computation allocation in dense-token processing, which introduces four key enhancements. First, we propose Dynamic Prompt Embedding (DPE) that adaptively processes only regions of interest while avoiding additional overhead from background tokens. Second, we introduce Dynamic Hybrid Attention (DHA), which leverages previous segmentation masks to route tokens through either full attention (O(N2)) for boundary regions or our proposed efficient BSQ attention (O(N)) for non-boundary regions. Third, we develop Hybrid Mixture of Experts (HMoE), which applies similar adaptive computation strategies in FFN modules with CPU-optimized parallel processing. Finally, we present Dynamic Local Upsampling (DLU), a reverse operation of DPE, which localizes objects with a lightweight MLP and performs fine-grained upsampling only in detected regions. Experimental results on high-precision IS benchmarks demonstrate that Inter2Former achieves SOTA performance with high efficiency on CPU devices.