Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation

TL;DR

This paper proposes a motion-aware contrastive learning framework to improve temporal scene graph generation by emphasizing motion patterns, outperforming existing methods on video and 4D datasets.

Contribution

It introduces a novel contrastive learning approach that leverages motion patterns to enhance temporal scene graph generation, addressing limitations of previous methods.

Findings

Significant performance improvements on video datasets.

Effective utilization of motion patterns in scene graph generation.

Outperforms state-of-the-art methods on multiple datasets.

Abstract

To equip artificial intelligence with a comprehensive understanding towards a temporal world, video and 4D panoptic scene graph generation abstracts visual data into nodes to represent entities and edges to capture temporal relations. Existing methods encode entity masks tracked across temporal dimensions (mask tubes), then predict their relations with temporal pooling operation, which does not fully utilize the motion indicative of the entities' relation. To overcome this limitation, we introduce a contrastive representation learning framework that focuses on motion pattern for temporal scene graph generation. Firstly, our framework encourages the model to learn close representations for mask tubes of similar subject-relation-object triplets. Secondly, we seek to push apart mask tubes from their temporally shuffled versions. Moreover, we also learn distant representations for mask tubes belonging to the same video but different triplets. Extensive experiments show that our motion-aware contrastive framework significantly improves state-of-the-art methods on both video and 4D datasets. Code is available at: https://github.com/nguyentthong/motion-contrastive-sgg