Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning

TL;DR

This paper introduces ISGR, a novel framework that enhances vision-language models' ability to reason about complex interactions in visual scenes by combining spatial relation extraction, interaction queries, and memory reinforcement learning.

Contribution

We propose Interaction-augmented Scene Graph Reasoning (ISGR), integrating spatial, interaction-aware, and memory components to improve scene understanding and reasoning in vision-language models.

Findings

Significantly outperforms baseline methods on interaction-heavy reasoning benchmarks.

Achieves strong improvements on complex scene understanding tasks.

Demonstrates effective long-term interaction reasoning capabilities.

Abstract

Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes. This paper addresses two key challenges: (1) conventional detection-to-construction methods produce unfocused, contextually irrelevant relationship sets, and (2) existing approaches fail to form persistent memories for generalizing interaction reasoning to new scenes. We propose Interaction-augmented Scene Graph Reasoning (ISGR), a framework that enhances VLMs' interactional reasoning through three complementary components. First, our dual-stream graph constructor combines SAM-powered spatial relation extraction with interaction-aware captioning to generate functionally salient scene graphs with spatial grounding. Second, we employ targeted interaction queries to activate VLMs' latent knowledge of object functionalities, converting passive recognition into active reasoning about how objects work together. Finally, we introduce a lone-term memory reinforcement learning strategy with a specialized interaction-focused reward function that transforms transient patterns into long-term reasoning heuristics. Extensive experiments demonstrate that our approach significantly outperforms baseline methods on interaction-heavy reasoning benchmarks, with particularly strong improvements on complex scene understanding tasks. The source code can be accessed at https://github.com/open_upon_acceptance.