Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

TL;DR

This paper introduces a tool-guided inference framework for vision-language models to better interpret optical illusions, achieving improved generalization without additional training.

Contribution

It proposes a generic-tool-plus-routing approach that enhances VLMs' ability to handle diverse illusions by using image manipulation tools and reasoning chains, without model training.

Findings

Performance remained consistent on unfamiliar illusion variants.

Identified a positive-detection bias linked to training data imbalance.

Observed a dissociation between spatial reasoning and logical inference.

Abstract

Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions: they overwhelmingly predict the illusion as "real" regardless of whether the image has been counterfactually modified. We present a tool-guided inference framework for the DataCV 2026 Challenge (Tasks I and II) that addresses this failure mode without any model training. An off-the-shelf vision-language model is given access to a small set of generic image manipulation tools: line drawing, region cropping, side-by-side comparison, and channel isolation, together with an illusion-type-routing system prompt that prescribes which tools to invoke for each perceptual question category. Critically, every tool call produces a new, immutable image resource appended to a persistent registry, so the model can reference and compose any prior annotated view throughout its reasoning chain. Rather than hard-coding illusion-specific modules, this generic-tool-plus-routing design yields strong cross-structural generalization: performance remained consistent from the validation set to a test set containing structurally unfamiliar illusion variants (e.g., Mach Bands rotated from vertical to horizontal stacking). We further report three empirical observations that we believe warrant additional investigation: (i) a strong positive-detection bias likely rooted in imbalanced illusion training data, (ii) a striking dissociation between pixel-accurate spatial reasoning and logical inference over self-generated annotations, and (iii) pronounced sensitivity to image compression artifacts that compounds false positives.