Towards Real Zero-Shot Camouflaged Object Segmentation without Camouflaged Annotations

TL;DR

This paper introduces a novel zero-shot camouflaged object segmentation framework that leverages semantic features and multimodal models, achieving state-of-the-art results without requiring any manual annotations.

Contribution

The proposed framework is the first to enable zero-shot COS using semantic transfer, multimodal alignment, and a learnable codebook for efficient inference.

Findings

Achieves $F_{eta}^w$ scores of 72.9% on CAMO and 71.7% on COD10K.

Runs at 18.1 FPS without the large language model during inference.

Outperforms existing methods in zero-shot camouflaged object segmentation.

Abstract

Camouflaged Object Segmentation (COS) faces significant challenges due to the scarcity of annotated data, where meticulous pixel-level annotation is both labor-intensive and costly, primarily due to the intricate object-background boundaries. Addressing the core question, "Can COS be effectively achieved in a zero-shot manner without manual annotations for any camouflaged object?" we affirmatively respond and introduce a robust zero-shot COS framework. This framework leverages the inherent local pattern bias of COS and employs a broad semantic feature space derived from salient object segmentation (SOS) for efficient zero-shot transfer. We incorporate an Masked Image Modeling (MIM) based image encoder optimized for Parameter-Efficient Fine-Tuning (PEFT), a Multimodal Large Language Model (M-LLM), and a Multi-scale Fine-grained Alignment (MFA) mechanism. The MIM pre-trained image encoder focuses on capturing essential low-level features, while the M-LLM generates caption embeddings processed alongside these visual cues. These embeddings are precisely aligned using MFA, enabling our framework to accurately interpret and navigate complex semantic contexts. To optimize operational efficiency, we introduce a learnable codebook that represents the M-LLM during inference, significantly reducing computational overhead. Our framework demonstrates its versatility and efficacy through rigorous experimentation, achieving state-of-the-art performance in zero-shot COS with $F_{\beta}^w$ scores of 72.9\% on CAMO and 71.7\% on COD10K. By removing the M-LLM during inference, we achieve an inference speed comparable to that of traditional end-to-end models, reaching 18.1 FPS. Code: https://github.com/AVC2-UESTC/ZSCOS-CaMF