What Secrets Do Your Manifolds Hold? Understanding the Local Geometry of Generative Models

TL;DR

This paper investigates the local geometric properties of learned data manifolds in deep generative models and demonstrates how these properties relate to generation quality, diversity, and memorization, enabling self-improvement through geometry-based rewards.

Contribution

It introduces a geometric framework with descriptors for analyzing local manifold structure and shows how these can guide improvements in generative model outputs.

Findings

Local geometry descriptors correlate with generation aesthetics and diversity

Training a reward model on local scaling improves generation quality

Geometry-based guidance enhances model self-optimization

Abstract

Deep Generative Models are frequently used to learn continuous representations of complex data distributions using a finite number of samples. For any generative model, including pre-trained foundation models with Diffusion or Transformer architectures, generation performance can significantly vary across the learned data manifold. In this paper we study the local geometry of the learned manifold and its relationship to generation outcomes for a wide range of generative models, including DDPM, Diffusion Transformer (DiT), and Stable Diffusion 1.4. Building on the theory of continuous piecewise-linear (CPWL) generators, we characterize the local geometry in terms of three geometric descriptors - scaling ($\psi$), rank ($\nu$), and complexity/un-smoothness ($\delta$). We provide quantitative and qualitative evidence showing that for a given latent-image pair, the local descriptors are indicative of generation aesthetics, diversity, and memorization by the generative model. Finally, we demonstrate that by training a reward model on the local scaling for Stable Diffusion, we can self-improve both generation aesthetics and diversity using `geometry reward' based guidance during denoising.