Recurrent Diffusion for Large-Scale Parameter Generation

TL;DR

Recurrent Diffusion for Large Scale Parameter Generation (RPG) is a new framework that efficiently generates large neural network parameters on a single GPU, enabling scalable and flexible model creation for various architectures and tasks.

Contribution

The paper introduces RPG, a novel recurrent diffusion-based method that scales parameter generation to hundreds of millions, surpassing previous memory and scalability limitations.

Findings

RPG achieves performance comparable to fully trained networks.

It generalizes to unseen tasks beyond its training set.

It enables large-scale parameter generation on a single GPU.

Abstract

Parameter generation has long struggled to match the scale of today large vision and language models, curbing its broader utility. In this paper, we introduce Recurrent Diffusion for Large Scale Parameter Generation (RPG), a novel framework that generates full neural network parameters up to hundreds of millions on a single GPU. Our approach first partitions a networks parameters into non-overlapping tokens, each corresponding to a distinct portion of the model. A recurrent mechanism then learns the inter token relationships, producing prototypes which serve as conditions for a diffusion process that ultimately synthesizes the full parameters. Across a spectrum of architectures and tasks including ResNets, ConvNeXts and ViTs on ImageNet 1K and COCO, and even LoRA based LLMs RPG achieves performance on par with fully trained networks while avoiding excessive memory overhead. Notably, it generalizes beyond its training set to generate valid parameters for previously unseen tasks, highlighting its flexibility in dynamic and open ended scenarios. By overcoming the longstanding memory and scalability barriers, RPG serves as a critical advance in AI generating AI, potentially enabling efficient weight generation at scales previously deemed infeasible.