Constrained Structure Learning for Scene Graph Generation

TL;DR

This paper introduces a novel constrained structure learning approach for scene graph generation, replacing traditional message passing with entropic mirror descent, leading to improved performance on benchmarks.

Contribution

It proposes a constrained variational inference framework using entropic mirror descent, offering a new inference strategy for scene graph generation beyond existing message passing methods.

Findings

Outperforms state-of-the-art methods on benchmarks

Validates the effectiveness of constrained optimization in scene graph generation

Demonstrates flexibility of the proposed inference approach

Abstract

As a structured prediction task, scene graph generation aims to build a visually-grounded scene graph to explicitly model objects and their relationships in an input image. Currently, the mean field variational Bayesian framework is the de facto methodology used by the existing methods, in which the unconstrained inference step is often implemented by a message passing neural network. However, such formulation fails to explore other inference strategies, and largely ignores the more general constrained optimization models. In this paper, we present a constrained structure learning method, for which an explicit constrained variational inference objective is proposed. Instead of applying the ubiquitous message-passing strategy, a generic constrained optimization method - entropic mirror descent - is utilized to solve the constrained variational inference step. We validate the proposed generic model on various popular scene graph generation benchmarks and show that it outperforms the state-of-the-art methods.