Text2Weight: Bridging Natural Language and Neural Network Weight Spaces

TL;DR

This paper introduces T2W, a diffusion transformer framework that generates neural network weights from natural language descriptions, improving generalization and enabling new applications in model customization.

Contribution

T2W is the first framework to generate task-specific neural network weights conditioned on natural language, combining hierarchical processing, CLIP-based text integration, and adversarial training.

Findings

Outperforms optimization-based initialization on unseen tasks

Enables weight enhancement and text-guided model fusion

Demonstrates high-quality weight generation on multiple datasets

Abstract

How far are we really from automatically generating neural networks? While neural network weight generation shows promise, current approaches struggle with generalization to unseen tasks and practical application exploration. To address this, we propose T2W, a diffusion transformer framework that generates task-specific weights conditioned on natural language descriptions. T2W hierarchically processes network parameters into uniform blocks, integrates text embeddings from CLIP via a prior attention mechanism, and employs adversarial training with weight-space augmentation to enhance generalization. Experiments on Cifar100, Caltech256, and TinyImageNet demonstrate T2W's ability to produce high-quality weights for unseen tasks, outperforming optimization-based initialization and enabling novel applications such as weight enhancement and text-guided model fusion. Our work bridges textual semantics with weight-space dynamics, supported by an open-source dataset of text-weight pairs, advancing the practicality of generative models in neural network parameter synthesis. Our code is available on Github.