Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency

TL;DR

This paper introduces CooK, a novel scene graph generation approach that incorporates object co-occurrence knowledge and learnable TF-IDF to improve relationship prediction, addressing long-tail issues and enhancing existing models.

Contribution

The paper proposes CooK, integrating co-occurrence knowledge and learnable TF-IDF into scene graph generation, significantly improving performance and generalization over state-of-the-art models.

Findings

Performance improved by up to 3.8% on SGGen subtask.

Generalization ability demonstrated across various MPNN models.

Effective handling of long-tail distribution in datasets.

Abstract

Scene graph generation (SGG) is an important task in image understanding because it represents the relationships between objects in an image as a graph structure, making it possible to understand the semantic relationships between objects intuitively. Previous SGG studies used a message-passing neural networks (MPNN) to update features, which can effectively reflect information about surrounding objects. However, these studies have failed to reflect the co-occurrence of objects during SGG generation. In addition, they only addressed the long-tail problem of the training dataset from the perspectives of sampling and learning methods. To address these two problems, we propose CooK, which reflects the Co-occurrence Knowledge between objects, and the learnable term frequency-inverse document frequency (TF-l-IDF) to solve the long-tail problem. We applied the proposed model to the SGG benchmark dataset, and the results showed a performance improvement of up to 3.8% compared with existing state-of-the-art models in SGGen subtask. The proposed method exhibits generalization ability from the results obtained, showing uniform performance improvement for all MPNN models.