Generative Image Modeling Using Spatial LSTMs

TL;DR

This paper introduces a novel spatial LSTM-based recurrent model for natural image generation that captures long-range dependencies, scales to arbitrary image sizes, and outperforms previous methods in quality and efficiency.

Contribution

The paper presents a new spatial LSTM-based recurrent image model that is scalable, computationally tractable, and superior to existing models in image generation tasks.

Findings

Outperforms state-of-the-art models in quantitative image dataset comparisons.

Produces high-quality texture synthesis and inpainting results.

Scales to images of arbitrary size with tractable likelihood computation.

Abstract

Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multi-dimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.