SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding

TL;DR

SGTA is a modular framework that combines scene graphs and multi-modal reasoning to interpret traffic videos, enabling accurate and interpretable question answering.

Contribution

It introduces a novel approach integrating structured scene graphs with large language models and tool-based reasoning for traffic video understanding.

Findings

SGTA achieves competitive accuracy on TUMTraffic VideoQA dataset.

It provides transparent reasoning steps for complex traffic video questions.

The framework effectively combines symbolic scene graphs with multi-modal inputs.

Abstract

We present Scene-Graph Based Multi-Modal Traffic Agent (SGTA), a modular framework for traffic video understanding that combines structured scene graphs with multi-modal reasoning. It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs. SGTA adopts ReAct to process interleaved reasoning traces from large language models with tool invocations, enabling interpretable decision-making for complex video questions. Experiments on selected TUMTraffic VideoQA dataset sample demonstrate that SGTA achieves competitive accuracy across multiple question types while providing transparent reasoning steps. These results highlight the potential of integrating structured scene representations with multi-modal agents for traffic video understanding.