Deep Structured Energy-Based Image Inpainting

TL;DR

This paper introduces a structured energy-based model for image inpainting that learns structural relationships between image patterns and missing regions, outperforming GAN-based methods on multiple benchmarks.

Contribution

The paper presents a novel energy-based structured prediction approach for image inpainting, demonstrating superior performance over existing GAN-based methods.

Findings

Achieved lower MSE on Olivetti face dataset

Higher PSNR on SVHN and CelebA datasets

Outperformed state-of-the-art GAN-based inpainting methods

Abstract

In this paper, we propose a structured image inpainting method employing an energy based model. In order to learn structural relationship between patterns observed in images and missing regions of the images, we employ an energy-based structured prediction method. The structural relationship is learned by minimizing an energy function which is defined by a simple convolutional neural network. The experimental results on various benchmark datasets show that our proposed method significantly outperforms the state-of-the-art methods which use Generative Adversarial Networks (GANs). We obtained 497.35 mean squared error (MSE) on the Olivetti face dataset compared to 833.0 MSE provided by the state-of-the-art method. Moreover, we obtained 28.4 dB peak signal to noise ratio (PSNR) on the SVHN dataset and 23.53 dB on the CelebA dataset, compared to 22.3 dB and 21.3 dB, provided by the state-of-the-art methods, respectively. The code is publicly available.