Position: Weight Space Should Be a First-Class Generative AI Modality

TL;DR

This paper advocates for treating neural network checkpoints as a primary data modality and standardizing generative modeling in weight space to enhance AI system development.

Contribution

It introduces a framework for viewing weight space as a generative modality, organizes existing methods, and highlights practical applications and current limitations.

Findings

Neural weights can be synthesized on demand with performance comparable to fine-tuning.

High-performing models occupy low-dimensional, structured regions of weight space.

Rapid progress in adapter-scale and conditional generation methods.

Abstract

Neural network checkpoints have quietly become a large-scale data resource: millions of trained weight vectors now exist, each encoding task-, domain-, and architecture-specific knowledge. This position paper argues that model checkpoints should be treated as a first-class data modality, and that generative modeling in weight space should be standardized as a core machine learning primitive. Recent advances demonstrate that neural weights can be synthesized on demand, often matching fine-tuning performance while reducing adaptation cost by orders of magnitude. We contend that these results reflect an underlying structural fact: high-performing models occupy low-dimensional, highly structured regions of weight space shaped by symmetry, flatness, modularity, and shared subspaces. Building on this view, we organize existing methods into a five-stage pipeline, survey applications where the approach is already practical, and clarify current limits: adapter-scale and conditional generation are advancing rapidly, while unrestricted frontier-scale checkpoint synthesis remains open. Our goal is to shift the community's default mindset from optimizing models per task to sampling models from learned weight distributions, accelerating toward an era in which AI systems routinely improve or create other AI systems.