Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

TL;DR

This paper introduces a novel method combining Markov Random Fields and deep convolutional neural networks to improve 2D image synthesis, enhancing realism and variability in generated images across photographic and artistic styles.

Contribution

It presents a new approach that integrates MRFs with dCNNs, enabling more plausible and diverse image synthesis than previous MRF or neural network methods.

Findings

Enhanced visual plausibility in photographic synthesis

Reduced artifacts and implausible feature mixtures

Greater variability in local feature matching

Abstract

This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthezing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.