From General to Specific: Informative Scene Graph Generation via Balance Adjustment

TL;DR

This paper introduces BA-SGG, a framework that improves scene graph generation by addressing imbalance issues between informative and common predicates, significantly enhancing performance across multiple sub-tasks.

Contribution

The paper proposes a novel balance adjustment framework with semantic and training sample components, applicable to various models for better predicate informativeness.

Findings

Achieves 14.3% higher mean recall on Visual Genome

Effectively adjusts semantic and sample imbalance

Improves state-of-the-art scene graph generation performance

Abstract

The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between informative predicates and common ones, i.e., semantic space level imbalance and training sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet effective SGG framework based on balance adjustment but not the conventional distribution fitting. It integrates two components: Semantic Adjustment (SA) and Balanced Predicate Learning (BPL), respectively for adjusting these imbalances. Benefited from the model-agnostic process, our method is easily applied to the state-of-the-art SGG models and significantly improves the SGG performance. Our method achieves 14.3%, 8.0%, and 6.1% higher Mean Recall (mR) than that of the Transformer model at three scene graph generation sub-tasks on Visual Genome, respectively. Codes are publicly available.