LinkNet: Relational Embedding for Scene Graph

TL;DR

This paper introduces LinkNet, a relational embedding model that enhances scene graph generation by modeling object interdependencies, achieving state-of-the-art results on the Visual Genome benchmark.

Contribution

The paper proposes a simple relational embedding module that jointly models object relationships, improving scene graph generation performance over existing methods.

Findings

Achieves state-of-the-art results on Visual Genome

Significantly improves relationship classification accuracy

Effective in modeling object inter-dependencies

Abstract

Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits the main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.