DFR: A Decompose-Fuse-Reconstruct Framework for Multi-Modal Few-Shot Segmentation

TL;DR

This paper introduces DFR, a framework that effectively combines visual, textual, and audio information for multi-modal few-shot segmentation, significantly improving performance by leveraging a novel decomposition, fusion, and reconstruction strategy.

Contribution

The paper proposes a new multi-modal framework that systematically integrates three modalities using hierarchical decomposition, contrastive fusion, and dual-path reconstruction for enhanced segmentation.

Findings

DFR outperforms existing methods on multiple benchmarks.

The hierarchical decomposition improves semantic extraction.

Contrastive fusion maintains cross-modal consistency.

Abstract

This paper presents DFR (Decompose, Fuse and Reconstruct), a novel framework that addresses the fundamental challenge of effectively utilizing multi-modal guidance in few-shot segmentation (FSS). While existing approaches primarily rely on visual support samples or textual descriptions, their single or dual-modal paradigms limit exploitation of rich perceptual information available in real-world scenarios. To overcome this limitation, the proposed approach leverages the Segment Anything Model (SAM) to systematically integrate visual, textual, and audio modalities for enhanced semantic understanding. The DFR framework introduces three key innovations: 1) Multi-modal Decompose: a hierarchical decomposition scheme that extracts visual region proposals via SAM, expands textual semantics into fine-grained descriptors, and processes audio features for contextual enrichment; 2) Multi-modal Contrastive Fuse: a fusion strategy employing contrastive learning to maintain consistency across visual, textual, and audio modalities while enabling dynamic semantic interactions between foreground and background features; 3) Dual-path Reconstruct: an adaptive integration mechanism combining semantic guidance from tri-modal fused tokens with geometric cues from multi-modal location priors. Extensive experiments across visual, textual, and audio modalities under both synthetic and real settings demonstrate DFR's substantial performance improvements over state-of-the-art methods.