ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images

TL;DR

ChromouVQA introduces a comprehensive benchmark for evaluating vision-language models on chromatic camouflaged images, highlighting current limitations and proposing a contrastive method to improve shape recognition in complex visual scenarios.

Contribution

This paper presents a new large-scale benchmark with multiple tasks and a contrastive approach to enhance shape recovery in camouflaged images, addressing a gap in multimodal understanding.

Findings

Humans outperform VLMs on chromatic camouflage tasks.

VLMs struggle with subtle chromatic contrasts and geometric disruptions.

Contrastive silhouette alignment improves shape recognition.

Abstract

Vision-Language Models (VLMs) have advanced multimodal understanding, yet still struggle when targets are embedded in cluttered backgrounds requiring figure-ground segregation. To address this, we introduce ChromouVQA, a large-scale, multi-task benchmark based on Ishihara-style chromatic camouflaged images. We extend classic dot plates with multiple fill geometries and vary chromatic separation, density, size, occlusion, and rotation, recording full metadata for reproducibility. The benchmark covers nine vision-question-answering tasks, including recognition, counting, comparison, and spatial reasoning. Evaluations of humans and VLMs reveal large gaps, especially under subtle chromatic contrast or disruptive geometric fills. We also propose a model-agnostic contrastive recipe aligning silhouettes with their camouflaged renderings, improving recovery of global shapes. ChromouVQA provides a compact, controlled benchmark for reproducible evaluation and extension. Code and dataset are available at https://github.com/Chromou-VQA-Benchmark/Chromou-VQA.