MOSA: Motion-Guided Semantic Alignment for Dynamic Scene Graph Generation

TL;DR

This paper introduces MoSA, a motion-guided semantic alignment method for dynamic scene graph generation, improving relationship modeling and tail relationship recognition in videos.

Contribution

It proposes a novel framework combining motion features, interaction modules, and semantic matching to enhance DSGG performance, especially for tail relationships.

Findings

MoSA achieves state-of-the-art results on the Action Genome dataset.

The motion-guided interaction module improves relationship representation.

The category-weighted loss emphasizes learning tail relationships.

Abstract

Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their dynamic interactions in video sequences for high-level semantic understanding. However, existing methods struggle with fine-grained relationship modeling, semantic representation utilization, and the ability to model tail relationships. To address these issues, this paper proposes a motion-guided semantic alignment method for DSGG (MoSA). First, a Motion Feature Extractor (MFE) encodes object-pair motion attributes such as distance, velocity, motion persistence, and directional consistency. Then, these motion attributes are fused with spatial relationship features through the Motion-guided Interaction Module (MIM) to generate motion-aware relationship representations. To further enhance semantic discrimination capabilities, the cross-modal Action Semantic Matching (ASM) mechanism aligns visual relationship features with text embeddings of relationship categories. Finally, a category-weighted loss strategy is introduced to emphasize learning of tail relationships. Extensive and rigorous testing shows that MoSA performs optimally on the Action Genome dataset.