Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

TL;DR

This paper investigates the robustness of vision-language models in reasoning tasks under distribution shifts, revealing limitations of current approaches and proposing a neuro-symbolic method, VLC, that enhances reasoning robustness across diverse tasks.

Contribution

The paper introduces VLC, a neuro-symbolic approach combining VLM-based concept recognition with circuit-based symbolic reasoning to improve robustness under distribution shifts.

Findings

VLMs perform well in-distribution but poorly under covariate shifts.

Neuro-symbolic approaches with black-box reasoning show inconsistent robustness.

VLC achieves strong, consistent performance across multiple reasoning tasks.

Abstract

Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC, a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves strong performance under covariate shifts, highlighting its ability to support robust reasoning.