Camouflage-aware Image-Text Retrieval via Expert Collaboration

TL;DR

This paper introduces a new camouflaged image-text retrieval task, creates a dedicated dataset, and proposes a collaborative network with a novel attention mechanism to improve retrieval accuracy in camouflaged scenarios.

Contribution

It formulates the first camouflaged image-text retrieval task, constructs a specialized dataset, and develops a novel collaborative network with confidence-conditioned graph attention.

Findings

CECNet achieves approximately 29% improvement in overall CA-ITR accuracy.

Benchmark results highlight the challenges posed by camouflage properties.

The proposed method surpasses seven existing retrieval models.

Abstract

Camouflaged scene understanding (CSU) has attracted significant attention due to its broad practical implications. However, in this field, robust image-text cross-modal alignment remains under-explored, hindering deeper understanding of camouflaged scenarios and their related applications. To this end, we focus on the typical image-text retrieval task, and formulate a new task dubbed ``camouflage-aware image-text retrieval'' (CA-ITR). We first construct a dedicated camouflage image-text retrieval dataset (CamoIT), comprising $\sim$10.5K samples with multi-granularity textual annotations. Benchmark results conducted on CamoIT reveal the underlying challenges of CA-ITR for existing cutting-edge retrieval techniques, which are mainly caused by objects' camouflage properties as well as those complex image contents. As a solution, we propose a camouflage-expert collaborative network (CECNet), which features a dual-branch visual encoder: one branch captures holistic image representations, while the other incorporates a dedicated model to inject representations of camouflaged objects. A novel confidence-conditioned graph attention (C\textsuperscript{2}GA) mechanism is incorporated to exploit the complementarity across branches. Comparative experiments show that CECNet achieves $\sim$29% overall CA-ITR accuracy boost, surpassing seven representative retrieval models. The dataset and code will be available at https://github.com/jiangyao-scu/CA-ITR.