Generative Escher Meshes

TL;DR

This paper introduces a fully-automatic, text-guided method for generating non-square, tileable 2D images with repeating patterns, optimizing both shape and texture through a differentiable mesh representation and diffusion-based guidance.

Contribution

It presents a novel differentiable framework for shape and texture optimization of tileable meshes guided by text prompts, enabling the creation of complex, non-square repeating patterns.

Findings

Produces plausible, appealing non-square tiles for various patterns

Utilizes a differentiable mesh parameterization and renderer

Leverages a diffusion model for text-based image guidance

Abstract

This paper proposes a fully-automatic, text-guided generative method for producing perfectly-repeating, periodic, tile-able 2D imagery, such as the one seen on floors, mosaics, ceramics, and the work of M.C. Escher. In contrast to square texture images that are seamless when tiled, our method generates non-square tilings which comprise solely of repeating copies of the same object. It achieves this by optimizing both geometry and texture of a 2D mesh, yielding a non-square tile in the shape and appearance of the desired object, with close to no additional background details, that can tile the plane without gaps nor overlaps. We enable optimization of the tile's shape by an unconstrained, differentiable parameterization of the space of all valid tileable meshes for given boundary conditions stemming from a symmetry group. Namely, we construct a differentiable family of linear systems derived from a 2D mesh-mapping technique - Orbifold Tutte Embedding - by considering the mesh's Laplacian matrix as differentiable parameters. We prove that the solution space of these linear systems is exactly all possible valid tiling configurations, thereby providing an end-to-end differentiable representation for the entire space of valid tiles. We render the textured mesh via a differentiable renderer, and leverage a pre-trained image diffusion model to induce a loss on the resulting image, updating the mesh's parameters so as to make its appearance match the text prompt. We show our method is able to produce plausible, appealing results, with non-trivial tiles, for a variety of different periodic tiling patterns.