Scaling Non-Parametric Sampling with Representation

TL;DR

This paper introduces a simple, non-parametric image generative model based on local pixel context, achieving high-quality samples without training and providing insights into natural image structure and generalization.

Contribution

The authors propose a minimal, training-free non-parametric model grounded in natural image principles, offering a transparent mechanism for image generation and understanding of generalization.

Findings

Produces high-fidelity MNIST samples

Generates visually compelling CIFAR-10 images

Reveals a simple part-whole generalization process

Abstract

Scaling and architectural advances have produced strikingly photorealistic image generative models, yet their mechanisms still remain opaque. Rather than advancing scaling, our goal is to strip away complicated engineering tricks and propose a simple, non-parametric generative model. Our design is grounded in three principles of natural images-(i) spatial non-stationarity, (ii) low-level regularities, and (iii) high-level semantics-and defines each pixel's distribution from its local context window. Despite its minimal architecture and no training, the model produces high-fidelity samples on MNIST and visually compelling CIFAR-10 images. This combination of simplicity and strong empirical performance points toward a minimal theory of natural-image structure. The model's white-box nature also allows us to have a mechanistic understanding of how the model generalizes and generates diverse images. We study it by tracing each generated pixel back to its source images. These analyses reveal a simple, compositional procedure for "part-whole generalization", suggesting a hypothesis for how large neural network generative models learn to generalize.