OED: Towards One-stage End-to-End Dynamic Scene Graph Generation

TL;DR

This paper introduces OED, a one-stage end-to-end framework for dynamic scene graph generation in videos, improving efficiency and temporal context modeling over traditional multi-stage methods.

Contribution

It proposes a novel set prediction approach with pair-wise features and a PRM module for better temporal dependency capture, enabling fully end-to-end training.

Findings

Outperforms existing methods on Action Genome benchmark

Effectively models temporal dependencies without additional trackers

Streamlines DSGG into a single end-to-end trainable framework

Abstract

Dynamic Scene Graph Generation (DSGG) focuses on identifying visual relationships within the spatial-temporal domain of videos. Conventional approaches often employ multi-stage pipelines, which typically consist of object detection, temporal association, and multi-relation classification. However, these methods exhibit inherent limitations due to the separation of multiple stages, and independent optimization of these sub-problems may yield sub-optimal solutions. To remedy these limitations, we propose a one-stage end-to-end framework, termed OED, which streamlines the DSGG pipeline. This framework reformulates the task as a set prediction problem and leverages pair-wise features to represent each subject-object pair within the scene graph. Moreover, another challenge of DSGG is capturing temporal dependencies, we introduce a Progressively Refined Module (PRM) for aggregating temporal context without the constraints of additional trackers or handcrafted trajectories, enabling end-to-end optimization of the network. Extensive experiments conducted on the Action Genome benchmark demonstrate the effectiveness of our design. The code and models are available at \url{https://github.com/guanw-pku/OED}.