DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs

TL;DR

DISSECT introduces a diagnostic benchmark to analyze the perception-integration gap in vision-language models, revealing differences between open- and closed-source models in visual reasoning capabilities.

Contribution

The paper presents DISSECT, a comprehensive diagnostic benchmark with a novel Model Oracle to systematically evaluate visual perception and reasoning in VLMs.

Findings

Chemistry questions are harder for visual reasoning than Biology.

Open-source models perform better with verbalized descriptions, indicating an integration bottleneck.

Closed-source models show no gap, suggesting better perception-integration bridging.

Abstract

When asked to describe a molecular diagram, a Vision-Language Model correctly identifies ``a benzene ring with an -OH group.'' When asked to reason about the same image, it answers incorrectly. The model can see but it cannot think about what it sees. We term this the perception-integration gap: a failure where visual information is successfully extracted but lost during downstream reasoning, invisible to single-configuration benchmarks that conflate perception with integration under one accuracy number. To systematically expose such failures, we introduce DISSECT, a 12,000-question diagnostic benchmark spanning Chemistry (7,000) and Biology (5,000). Every question is evaluated under five input modes -- Vision+Text, Text-Only, Vision-Only, Human Oracle, and a novel Model Oracle in which the VLM first verbalizes the image and then reasons from its own description -- yielding diagnostic gaps that decompose performance into language-prior exploitation, visual extraction, perception fidelity, and integration effectiveness. Evaluating 18~VLMs, we find that: (1) Chemistry exhibits substantially lower language-prior exploitability than Biology, confirming molecular visual content as a harder test of genuine visual reasoning; (2) Open-source models consistently score higher when reasoning from their own verbalized descriptions than from raw images, exposing a systematic integration bottleneck; and (3) Closed-source models show no such gap, indicating that bridging perception and integration is the frontier separating open-source from closed-source multimodal capability. The Model Oracle protocol is both model and benchmark agnostic, applicable post-hoc to any VLM evaluation to diagnose integration failures.