Open-Vocabulary Camouflaged Object Segmentation with Cascaded Vision Language Models

TL;DR

This paper introduces a VLM-guided cascaded framework for open-vocabulary camouflaged object segmentation that improves localization and classification accuracy by leveraging rich semantics and explicit prompts, addressing domain gaps and boundary issues.

Contribution

The paper proposes a novel VLM-guided cascaded approach that integrates SAM with VLM-derived features for improved camouflaged object segmentation and classification.

Findings

Significantly better segmentation accuracy on OVCOS benchmarks.

Enhanced classification performance with full-image context.

Effective use of VLM semantics for both segmentation and classification.

Abstract

Open-Vocabulary Camouflaged Object Segmentation (OVCOS) seeks to segment and classify camouflaged objects from arbitrary categories, presenting unique challenges due to visual ambiguity and unseen categories.Recent approaches typically adopt a two-stage paradigm: first segmenting objects, then classifying the segmented regions using Vision Language Models (VLMs).However, these methods (1) suffer from a domain gap caused by the mismatch between VLMs' full-image training and cropped-region inference, and (2) depend on generic segmentation models optimized for well-delineated objects, making them less effective for camouflaged objects.Without explicit guidance, generic segmentation models often overlook subtle boundaries, leading to imprecise segmentation.In this paper,we introduce a novel VLM-guided cascaded framework to address these issues in OVCOS.For segmentation, we leverage the Segment Anything Model (SAM), guided by the VLM.Our framework uses VLM-derived features as explicit prompts to SAM, effectively directing attention to camouflaged regions and significantly improving localization accuracy.For classification, we avoid the domain gap introduced by hard cropping.Instead, we treat the segmentation output as a soft spatial prior via the alpha channel, which retains the full image context while providing precise spatial guidance, leading to more accurate and context-aware classification of camouflaged objects.The same VLM is shared across both segmentation and classification to ensure efficiency and semantic consistency.Extensive experiments on both OVCOS and conventional camouflaged object segmentation benchmarks demonstrate the clear superiority of our method, highlighting the effectiveness of leveraging rich VLM semantics for both segmentation and classification of camouflaged objects.