Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

TL;DR

This paper introduces Pair Affinity Learning and Scoring (PALS) with Relation-Aware Matching (RAM) to improve weakly-supervised video scene graph generation by filtering noninteractive object pairs, leading to state-of-the-art results.

Contribution

It proposes a novel pair affinity estimation method and a relation-aware pseudo-labeling technique to enhance weakly-supervised scene graph generation in videos.

Findings

Achieves state-of-the-art performance on Action Genome dataset.

Significantly improves relation detection accuracy.

Effectively suppresses noninteractive object pairs.

Abstract

Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling, significantly reducing annotation costs. Without ground-truth bounding boxes, these methods rely on off-the-shelf detectors to generate object proposals, yet largely overlook a fundamental discrepancy from fullysupervised pipelines. Fully-supervised detectors implicitly filter out noninteractive objects, while off-the-shelf detectors indiscriminately detect all visible objects, overwhelming relation models with noisy pairs.We address this by introducing a learnable pair affinity that estimates the likelihood of interaction between subject-object pairs. Through Pair Affinity Learning and Scoring (PALS), pair affinity is incorporated into inferencetime ranking and further integrated into contextual reasoning through Pair Affinity Modulation (PAM), enabling the model to suppress noninteractive pairs and focus on relationally meaningful ones. To provide cleaner supervision for pair affinity learning, we further propose Relation- Aware Matching (RAM), which leverages vision-language grounding to resolve class-level ambiguity in pseudo-label generation. Extensive experiments on Action Genome demonstrate that our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance.