SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

TL;DR

SceneGraphVLM introduces a compact, efficient method for generating scene graphs from images and videos using small vision-language models, balancing quality and speed.

Contribution

It proposes a two-stage training approach with hallucination-aware rewards and a token-efficient graph serialization format for improved scene graph generation.

Findings

Achieves a strong quality-speed trade-off with approximately one-second latency.

Improves precision-oriented scene graph metrics while maintaining reasonable recall.

Supports conditioning on previous frames for lightweight video scene graph generation.

Abstract

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.