A Generative Model for Texture Synthesis based on Optimal Transport between Feature Distributions

TL;DR

GOTEX is a versatile framework for texture synthesis using optimal transport to match feature distributions, enabling high-quality, flexible, and fast texture generation and inpainting.

Contribution

It introduces a Wasserstein-based generative model for texture synthesis that controls feature distributions via optimal transport, adaptable to various features and architectures.

Findings

Produces high-quality textures with different feature sets

Outperforms state-of-the-art methods in quality and flexibility

Enables fast, on-the-fly texture synthesis with learned neural networks

Abstract

We propose GOTEX, a general framework for texture synthesis by optimization that constrains the statistical distribution of local features. While our model encompasses several existing texture models, we focus on the case where the comparison between feature distributions relies on optimal transport distances. We show that the semi-dual formulation of optimal transport allows to control the distribution of various possible features, even if these features live in a high-dimensional space. We then study the resulting minimax optimization problem, which corresponds to a Wasserstein generative model, for which the inner concave maximization problem can be solved with standard stochastic gradient methods. The alternate optimization algorithm is shown to be versatile in terms of applications, features and architecture; in particular it allows to produce high-quality synthesized textures with different sets of features. We analyze the results obtained by constraining the distribution of patches or the distribution of responses to a pre-learned VGG neural network. We show that the patch representation can retrieve the desired textural aspect in a more precise manner. We also provide a detailed comparison with state-of-the-art texture synthesis methods. The GOTEX model based on patch features is also adapted to texture inpainting and texture interpolation. Finally, we show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size in a very fast manner. Experimental results and comparisons with the mainstream methods from the literature illustrate the relevance of the generative models learned with GOTEX.