Taking A Closer Look at Interacting Objects: Interaction-Aware Open Vocabulary Scene Graph Generation

TL;DR

This paper introduces INOVA, an interaction-aware framework for open vocabulary scene graph generation that explicitly models object interactions to improve relation prediction and overall performance.

Contribution

INOVA is the first to incorporate explicit interaction modeling in open vocabulary scene graph generation, improving relation accuracy and robustness.

Findings

Achieves state-of-the-art results on VG and GQA benchmarks.

Effectively distinguishes interacting objects from non-interacting ones.

Enhances robustness through interaction-consistent knowledge distillation.

Abstract

Today's open vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Most existing methods adopt a two-stage pipeline: weakly supervised pre-training with image captions and supervised fine-tuning (SFT) on fully annotated scene graphs. Nonetheless, they omit explicit modeling of interacting objects and treat all objects equally, resulting in mismatched relation pairs. To this end, we propose an interaction-aware OVSGG framework INOVA. During pre-training, INOVA employs an interaction-aware target generation strategy to distinguish interacting objects from non-interacting ones. In SFT, INOVA devises an interaction-guided query selection tactic to prioritize interacting objects during bipartite graph matching. Besides, INOVA is equipped with an interaction-consistent knowledge distillation to enhance the robustness by pushing interacting object pairs away from the background. Extensive experiments on two benchmarks (VG and GQA) show that INOVA achieves state-of-the-art performance, demonstrating the potential of interaction-aware mechanisms for real-world applications.