VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation

TL;DR

This paper introduces VOST-SGG, a novel one-stage spatio-temporal scene graph generation framework that leverages vision-language models to incorporate semantic priors and multi-modal features, significantly improving performance.

Contribution

The paper proposes a VLM-aided approach with dual-source query initialization and multi-modal feature fusion for enhanced ST-SGG.

Findings

Achieves state-of-the-art results on Action Genome dataset

Demonstrates the effectiveness of semantic priors from VLMs

Validates multi-modal features improve predicate classification

Abstract

Spatio-temporal scene graph generation (ST-SGG) aims to model objects and their evolving relationships across video frames, enabling interpretable representations for downstream reasoning tasks such as video captioning and visual question answering. Despite recent advancements in DETR-style single-stage ST-SGG models, they still suffer from several key limitations. First, while these models rely on attention-based learnable queries as a core component, these learnable queries are semantically uninformed and instance-agnostically initialized. Second, these models rely exclusively on unimodal visual features for predicate classification. To address these challenges, we propose VOST-SGG, a VLM-aided one-stage ST-SGG framework that integrates the common sense reasoning capabilities of vision-language models (VLMs) into the ST-SGG pipeline. First, we introduce the dual-source query initialization strategy that disentangles what to attend to from where to attend, enabling semantically grounded what-where reasoning. Furthermore, we propose a multi-modal feature bank that fuses visual, textual, and spatial cues derived from VLMs for improved predicate classification. Extensive experiments on the Action Genome dataset demonstrate that our approach achieves state-of-the-art performance, validating the effectiveness of integrating VLM-aided semantic priors and multi-modal features for ST-SGG. We will release the code at https://github.com/LUNAProject22/VOST.