Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click

TL;DR

Click2Graph is an interactive framework that allows users to generate and understand panoptic video scene graphs from minimal input, combining human guidance with advanced visual and relational reasoning.

Contribution

It introduces the first interactive system for panoptic video scene graph generation that integrates user prompts with semantic and relational understanding.

Findings

Outperforms existing methods on the OpenPVSG benchmark.

Enables real-time user-guided scene graph generation.

Demonstrates effective integration of human prompts with automated reasoning.

Abstract

State-of-the-art Video Scene Graph Generation (VSGG) systems provide structured visual understanding but operate as closed, feed-forward pipelines with no ability to incorporate human guidance. In contrast, promptable segmentation models such as SAM2 enable precise user interaction but lack semantic or relational reasoning. We introduce Click2Graph, the first interactive framework for Panoptic Video Scene Graph Generation (PVSG) that unifies visual prompting with spatial, temporal, and semantic understanding. From a single user cue, such as a click or bounding box, Click2Graph segments and tracks the subject across time, autonomously discovers interacting objects, and predicts <subject, object, predicate> triplets to form a temporally consistent scene graph. Our framework introduces two key components: a Dynamic Interaction Discovery Module that generates subject-conditioned object prompts, and a Semantic Classification Head that performs joint entity and predicate reasoning. Experiments on the OpenPVSG benchmark demonstrate that Click2Graph establishes a strong foundation for user-guided PVSG, showing how human prompting can be combined with panoptic grounding and relational inference to enable controllable and interpretable video scene understanding.