Texture Mixing by Interpolating Deep Statistics via Gaussian Models

TL;DR

This paper introduces a Gaussian-based interpolation method for mixing textures using deep neural network statistics, enabling improved style transfer with minimal additional computation.

Contribution

It unifies deep texture statistics with a Gaussian scheme and proposes an optimal transport-based interpolation method for texture mixing.

Findings

Achieves state-of-the-art texture mixing results

Efficient closed-form computations with negligible extra time

Effective application to Neural Style Transfer

Abstract

Recently, enthusiastic studies have devoted to texture synthesis using deep neural networks, because these networks excel at handling complex patterns in images. In these models, second-order statistics, such as Gram matrix, are used to describe textures. Despite the fact that these model have achieved promising results, the structure of their parametric space is still unclear, consequently, it is difficult to use them to mix textures. This paper addresses the texture mixing problem by using a Gaussian scheme to interpolate deep statistics computed from deep neural networks. More precisely, we first reveal that the statistics used in existing deep models can be unified using a stationary Gaussian scheme. We then present a novel algorithm to mix these statistics by interpolating between Gaussian models using optimal transport. We further apply our scheme to Neural Style Transfer, where we can create mixed styles. The experiments demonstrate that our method can achieve state-of-the-art results. Because all the computations are implemented in closed forms, our mixing algorithm adds only negligible time to the original texture synthesis procedure.