Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

TL;DR

This paper introduces a framework for evaluating and enhancing vision-language models' ability to perform compositional reasoning, using structural priors and scene graph parsing to improve accuracy on the Winoground benchmark.

Contribution

It presents a unified evaluation and augmentation framework that incorporates structural relational priors via scene graph parsing to improve compositional reasoning in VLMs.

Findings

Qwen3-VL-8B-Thinking achieves a group score of 62.75, surpassing other models.

Multi-turn scene graph filtering raises the score to 66.0, surpassing prior state-of-the-art.

Scene graph augmentation benefits capable models but not weaker baselines.

Abstract

Vision-language models (VLMs) excel at image-text retrieval yet persistently fail at compositional reasoning, distinguishing captions that share the same words but differ in relational structure. We present, a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs,CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking,on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject-relation-object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject-object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.75, far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0, surpassing prior open-source state-of-the-art. We analyze the capability augmentation tradeoff and find that SG augmentation benefits already capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding