Doubly Reparameterized Importance Weighted Structure Learning for Scene Graph Generation

TL;DR

This paper introduces a novel importance weighted structure learning approach for scene graph generation, employing a tighter variational bound and reparameterization techniques to improve accuracy and reduce variance, achieving state-of-the-art results.

Contribution

It proposes a doubly reparameterized importance weighted variational inference method with a Gumbel-Softmax sampler for scene graph generation, enhancing posterior approximation.

Findings

Achieves state-of-the-art performance on scene graph benchmarks.

Reduces variance of gradient estimates, improving learning stability.

Employs a tighter variational lower bound for better posterior modeling.

Abstract

As a structured prediction task, scene graph generation, given an input image, aims to explicitly model objects and their relationships by constructing a visually-grounded scene graph. In the current literature, such task is universally solved via a message passing neural network based mean field variational Bayesian methodology. The classical loose evidence lower bound is generally chosen as the variational inference objective, which could induce oversimplified variational approximation and thus underestimate the underlying complex posterior. In this paper, we propose a novel doubly reparameterized importance weighted structure learning method, which employs a tighter importance weighted lower bound as the variational inference objective. It is computed from multiple samples drawn from a reparameterizable Gumbel-Softmax sampler and the resulting constrained variational inference task is solved by a generic entropic mirror descent algorithm. The resulting doubly reparameterized gradient estimator reduces the variance of the corresponding derivatives with a beneficial impact on learning. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.