Unbiased Visual Reasoning with Controlled Visual Inputs

TL;DR

VISTA is a modular framework that improves unbiased visual reasoning by separating perception from reasoning, using an explicit information bottleneck and reinforcement learning to reduce reliance on spurious correlations.

Contribution

It introduces VISTA, a novel approach that decouples perception and reasoning in vision-language models, enhancing robustness and interpretability in visual question answering.

Findings

Significantly improves robustness to spurious correlations on SpuriVerse.

Transfers robustly across unseen VLM sensors.

Produces more neutral and visually grounded reasoning traces.

Abstract

End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset. VISTA transfers robustly across unseen VLM sensors and is able to recognize and recover from VLM perception failures. Human analysis further shows that VISTA's reasoning traces are more neutral, less reliant on spurious attributes, and more explicitly grounded in visual evidence than end-to-end VLM baselines.