Multi-objects Generation with Amortized Structural Regularization

TL;DR

This paper introduces the amortized structural regularization framework that incorporates human knowledge into deep generative models using structural constraints, improving multi-object image generation quality.

Contribution

It proposes a novel amortized structural regularization method that embeds human knowledge into DGMs via posterior regularization for better multi-object image modeling.

Findings

ASR outperforms baseline DGMs in inference accuracy.

ASR improves sample quality in multi-object image generation.

The approach efficiently optimizes a lower bound of regularized log-likelihood.

Abstract

Deep generative models (DGMs) have shown promise in image generation. However, most of the existing work learn the model by simply optimizing a divergence between the marginal distributions of the model and the data, and often fail to capture the rich structures and relations in multi-object images. Human knowledge is a critical element to the success of DGMs to infer these structures. In this paper, we propose the amortized structural regularization (ASR) framework, which adopts the posterior regularization (PR) to embed human knowledge into DGMs via a set of structural constraints. We derive a lower bound of the regularized log-likelihood, which can be jointly optimized with respect to the generative model and recognition model efficiently. Empirical results show that ASR significantly outperforms the DGM baselines in terms of inference accuracy and sample quality.