Neural Belief Propagation for Scene Graph Generation

TL;DR

This paper introduces a neural belief propagation approach for scene graph generation that models higher-order interactions, leading to more consistent interpretations and state-of-the-art results.

Contribution

It proposes a novel neural belief propagation method using Bethe approximation, incorporating higher-order interactions for improved scene graph accuracy.

Findings

Achieves state-of-the-art performance on scene graph benchmarks.

Models higher-order interactions for more consistent scene interpretations.

Outperforms previous message passing neural network models.

Abstract

Scene graph generation aims to interpret an input image by explicitly modelling the potential objects and their relationships, which is predominantly solved by the message passing neural network models in previous methods. Currently, such approximation models generally assume the output variables are totally independent and thus ignore the informative structural higher-order interactions. This could lead to the inconsistent interpretations for an input image. In this paper, we propose a novel neural belief propagation method to generate the resulting scene graph. It employs a structural Bethe approximation rather than the mean field approximation to infer the associated marginals. To find a better bias-variance trade-off, the proposed model not only incorporates pairwise interactions but also higher order interactions into the associated scoring function. It achieves the state-of-the-art performance on various popular scene graph generation benchmarks.