Scene Graph Modification as Incremental Structure Expanding

TL;DR

This paper introduces an incremental approach to scene graph modification that updates existing graphs based on natural language queries, improving accuracy and efficiency over previous methods.

Contribution

The paper proposes a novel incremental structure expanding method for scene graph modification, avoiding complete rebuilding and enhancing prediction accuracy.

Findings

Outperforms previous state-of-the-art models on four benchmarks.

Constructs a challenging new dataset with complex queries and larger graphs.

Demonstrates the effectiveness of incremental expansion in scene graph modification.

Abstract

A scene graph is a semantic representation that expresses the objects, attributes, and relationships between objects in a scene. Scene graphs play an important role in many cross modality tasks, as they are able to capture the interactions between images and texts. In this paper, we focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. Unlike previous approaches that rebuilt the entire scene graph, we frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE). ISE constructs the target graph by incrementally expanding the source graph without changing the unmodified structure. Based on ISE, we further propose a model that iterates between nodes prediction and edges prediction, inferring more accurate and harmonious expansion decisions progressively. In addition, we construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets. Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins.