Temporally Consistent Dynamic Scene Graphs: An End-to-End Approach for Action Tracklet Generation

TL;DR

This paper introduces TCDSG, an end-to-end framework for generating temporally consistent dynamic scene graphs that track and link object interactions over time, significantly improving action tracklet generation in videos.

Contribution

It presents a novel bipartite matching mechanism with adaptive decoder queries for robust, temporally coherent scene graph generation across video sequences.

Findings

Achieves over 60% improvement in temporal recall@k on multiple datasets.

Establishes a new benchmark for dynamic scene graph generation.

Augments MEVA dataset with persistent object ID annotations.

Abstract

Understanding video content is pivotal for advancing real-world applications like activity recognition, autonomous systems, and human-computer interaction. While scene graphs are adept at capturing spatial relationships between objects in individual frames, extending these representations to capture dynamic interactions across video sequences remains a significant challenge. To address this, we present TCDSG, Temporally Consistent Dynamic Scene Graphs, an innovative end-to-end framework that detects, tracks, and links subject-object relationships across time, generating action tracklets, temporally consistent sequences of entities and their interactions. Our approach leverages a novel bipartite matching mechanism, enhanced by adaptive decoder queries and feedback loops, ensuring temporal coherence and robust tracking over extended sequences. This method not only establishes a new benchmark by achieving over 60% improvement in temporal recall@k on the Action Genome, OpenPVSG, and MEVA datasets but also pioneers the augmentation of MEVA with persistent object ID annotations for comprehensive tracklet generation. By seamlessly integrating spatial and temporal dynamics, our work sets a new standard in multi-frame video analysis, opening new avenues for high-impact applications in surveillance, autonomous navigation, and beyond.