PixelTransformer: Sample Conditioned Signal Generation

TL;DR

PixelTransformer is a novel generative model that infers spatial signal distributions conditioned on sparse samples, enabling diverse, meaningful outputs across various data types and improving with more observed data.

Contribution

It introduces a flexible, sample-conditioned generative model that surpasses autoregressive methods by handling arbitrary conditioning and distributional queries for spatial data.

Findings

Generates diverse, meaningful images from sparse samples

Distribution variance decreases with more observed pixels

Applicable to various spatial data types beyond images

Abstract

We propose a generative model that can infer a distribution for the underlying spatial signal conditioned on sparse samples e.g. plausible images given a few observed pixels. In contrast to sequential autoregressive generative models, our model allows conditioning on arbitrary samples and can answer distributional queries for any location. We empirically validate our approach across three image datasets and show that we learn to generate diverse and meaningful samples, with the distribution variance reducing given more observed pixels. We also show that our approach is applicable beyond images and can allow generating other types of spatial outputs e.g. polynomials, 3D shapes, and videos.