Hierarchical Memory Learning for Fine-Grained Scene Graph Generation

TL;DR

This paper introduces a Hierarchical Memory Learning framework for Scene Graph Generation that mimics human hierarchical learning, improving predicate prediction by addressing data imbalance and granularity issues.

Contribution

It proposes a novel HML framework with Concept and Model Reconstruction constraints, enabling hierarchical learning from coarse to fine predicates in SGG.

Findings

Significant performance improvements on Visual Genome benchmark.

Effective alleviation of long-tail and mixed-granularity problems.

Framework applicable to various SGG models.

Abstract

As far as Scene Graph Generation (SGG), coarse and fine predicates mix in the dataset due to the crowd-sourced labeling, and the long-tail problem is also pronounced. Given this tricky situation, many existing SGG methods treat the predicates equally and learn the model under the supervision of mixed-granularity predicates in one stage, leading to relatively coarse predictions. In order to alleviate the negative impact of the suboptimum mixed-granularity annotation and long-tail effect problems, this paper proposes a novel Hierarchical Memory Learning (HML) framework to learn the model from simple to complex, which is similar to the human beings' hierarchical memory learning process. After the autonomous partition of coarse and fine predicates, the model is first trained on the coarse predicates and then learns the fine predicates. In order to realize this hierarchical learning pattern, this paper, for the first time, formulates the HML framework using the new Concept Reconstruction (CR) and Model Reconstruction (MR) constraints. It is worth noticing that the HML framework can be taken as one general optimization strategy to improve various SGG models, and significant improvement can be achieved on the SGG benchmark (i.e., Visual Genome).