SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation

TL;DR

SceneLLM introduces a novel framework leveraging large language models for dynamic scene graph generation by transforming video data into linguistic signals, encoding spatial info, and fine-tuning for improved scene understanding.

Contribution

The paper presents SceneLLM, a new approach that uses LLMs with a Video-to-Language module, spatial encoding, and implicit language signals for dynamic scene graph generation.

Findings

Achieves state-of-the-art results on Action Genome benchmark.

Effectively encodes spatio-temporal information into linguistic signals.

Demonstrates improved scene understanding and graph accuracy.

Abstract

Dynamic scenes contain intricate spatio-temporal information, crucial for mobile robots, UAVs, and autonomous driving systems to make informed decisions. Parsing these scenes into semantic triplets <Subject-Predicate-Object> for accurate Scene Graph Generation (SGG) is highly challenging due to the fluctuating spatio-temporal complexity. Inspired by the reasoning capabilities of Large Language Models (LLMs), we propose SceneLLM, a novel framework that leverages LLMs as powerful scene analyzers for dynamic SGG. Our framework introduces a Video-to-Language (V2L) mapping module that transforms video frames into linguistic signals (scene tokens), making the input more comprehensible for LLMs. To better encode spatial information, we devise a Spatial Information Aggregation (SIA) scheme, inspired by the structure of Chinese characters, which encodes spatial data into tokens. Using Optimal Transport (OT), we generate an implicit language signal from the frame-level token sequence that captures the video's spatio-temporal information. To further improve the LLM's ability to process this implicit linguistic input, we apply Low-Rank Adaptation (LoRA) to fine-tune the model. Finally, we use a transformer-based SGG predictor to decode the LLM's reasoning and predict semantic triplets. Our method achieves state-of-the-art results on the Action Genome (AG) benchmark, and extensive experiments show the effectiveness of SceneLLM in understanding and generating accurate dynamic scene graphs.