Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis

TL;DR

This paper challenges the assumption that global data distribution learning suffices for image generation, proposing new frameworks to analyze the transition from generative behavior to memorization in probabilistic models.

Contribution

It introduces the Mutually Exclusive Probability Space and Local Dependence Hypothesis frameworks, along with the $oldsymbol{ extgamma}$-Autoregressive Random Variable Model, to analyze and understand memorization in image generation.

Findings

Increasing observation range leads models toward memorization.

Binary Latent Autoencoder encodes images into signed binary representations.

Models behave as pure memorizer at global dependence limit.

Abstract

A common assumption in probabilistic generative models for image generation is that learning the global data distribution suffices to generate novel images via sampling. We investigate the limitation of this core assumption, namely that learning global distributions leads to memorization rather than generative behavior. We propose two theoretical frameworks, the Mutually Exclusive Probability Space (MEPS) and the Local Dependence Hypothesis (LDH), for investigation. MEPS arises from the observation that deterministic mappings (e.g. neural networks) involving random variables tend to reduce overlap coefficients among involved random variables, thereby inducing exclusivity. We further propose a lower bound in terms of the overlap coefficient, and introduce a Binary Latent Autoencoder (BL-AE) that encodes images into signed binary latent representations. LDH formalizes dependence within a finite observation radius, which motivates our $\gamma$-Autoregressive Random Variable Model ($\gamma$-ARVM). $\gamma$-ARVM is an autoregressive model, with a variable observation range $\gamma$, that predicts a histogram for the next token. Using $\gamma$-ARVM, we observe that as the observation range increases, autoregressive models progressively shift toward memorization. In the limit of global dependence, the model behaves as a pure memorizer when operating on the binary latents produced by our BL-AE. Comprehensive experiments and discussions support our investigation.