Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation

TL;DR

This paper introduces ImparTail, a training framework that reduces bias and improves robustness in spatio-temporal scene graph generation and anticipation by using loss masking, curriculum learning, and new benchmark tasks.

Contribution

We propose a novel unbiased training method with curriculum-driven mask generation for more balanced spatio-temporal scene graph models, along with new robustness benchmarks.

Findings

Outperforms existing methods in unbiased scene graph generation

Demonstrates robustness under distribution shifts

Achieves superior results on the Action Genome dataset

Abstract

Spatio-Temporal Scene Graphs (STSGs) provide a concise and expressive representation of dynamic scenes by modeling objects and their evolving relationships over time. However, real-world visual relationships often exhibit a long-tailed distribution, causing existing methods for tasks like Video Scene Graph Generation (VidSGG) and Scene Graph Anticipation (SGA) to produce biased scene graphs. To this end, we propose ImparTail, a novel training framework that leverages loss masking and curriculum learning to mitigate bias in the generation and anticipation of spatio-temporal scene graphs. Unlike prior methods that add extra architectural components to learn unbiased estimators, we propose an impartial training objective that reduces the dominance of head classes during learning and focuses on underrepresented tail relationships. Our curriculum-driven mask generation strategy further empowers the model to adaptively adjust its bias mitigation strategy over time, enabling more balanced and robust estimations. To thoroughly assess performance under various distribution shifts, we also introduce two new tasks Robust Spatio-Temporal Scene Graph Generation and Robust Scene Graph Anticipation offering a challenging benchmark for evaluating the resilience of STSG models. Extensive experiments on the Action Genome dataset demonstrate the superior unbiased performance and robustness of our method compared to existing baselines.