Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning

TL;DR

This paper introduces Structured Qualitative Inference (SQI), a training-free framework that enhances the perceptual robustness of frozen Vision-Language Models against optical illusions by integrating systematic qualitative reasoning modules.

Contribution

The paper presents a novel, data-centric approach with three modules—axiomatic constraints, hierarchical scene decomposition, and counterfactual verification—that improve visual grounding without model fine-tuning.

Findings

SQI ranked 2nd in the DataCV 2026 Challenge for illusion understanding.

Experimental results show significant accuracy improvements across illusion categories.

SQI offers interpretability benefits without requiring model fine-tuning.

Abstract

While Vision-Language Models (VLMs) have achieved state-of-the-art performance in general visual tasks, their perceptual robustness remains remarkably brittle when confronted with optical illusions. These failures are often attributed to shortcut heuristics, where models prioritize linguistic priors and memorized prototypes over direct visual evidence. In this work, we propose Structured Qualitative Inference (SQI), a training-free, data-centric framework designed to fortify visual grounding in frozen VLMs. SQI addresses perceptual anomalies through three systematic modules: (1) Axiomatic Constraint Injection, which suppresses erroneous metric estimations and quantitative hallucinations; (2) Hierarchical Scene Decomposition, which decouples target visual manifolds from complex background distractors; and (3) Counterfactual Self-Verification, an adversarial reasoning step that mitigates confirmation bias. By orchestrating these qualitative constraints at inference time, SQI effectively aligns high-level linguistic reasoning with low-level visual perception. Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall. Experimental results demonstrate that SQI not only significantly enhances accuracy across diverse illusion categories but also provides superior diagnostic interpretability without any model fine-tuning. Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.