Pluralistic Image Completion with Probabilistic Mixture-of-Experts

TL;DR

This paper introduces a probabilistic graph model with a Gaussian mixture component to generate diverse, realistic image completions, addressing interpretability and diversity issues of prior methods.

Contribution

It proposes an end-to-end probabilistic approach with a GMM to better enforce constraints and control diversity in pluralistic image completion.

Findings

Outperforms prior methods in visual realism and diversity

Effectively controls diversity via GMM primitives

Demonstrates superior results through comprehensive experiments

Abstract

Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints to not work well. In this paper, to address the issues, we propose an end-to-end probabilistic method. Specifically, we introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments.