Diffusion-Based Neural Network Weights Generation

TL;DR

This paper introduces D2NWG, a diffusion-based method for generating neural network weights conditioned on target datasets, improving transfer learning efficiency and performance across various models, including large language models.

Contribution

The paper presents a novel diffusion-based approach for neural network weight generation that generalizes across tasks and models, surpassing existing meta-learning and pretrained methods.

Findings

Outperforms state-of-the-art meta-learning methods.

Scalable to large architectures like LLMs.

Enhances performance of diverse base models.

Abstract

Transfer learning has gained significant attention in recent deep learning research due to its ability to accelerate convergence and enhance performance on new tasks. However, its success is often contingent on the similarity between source and target data, and training on numerous datasets can be costly, leading to blind selection of pretrained models with limited insight into their effectiveness. To address these challenges, we introduce D2NWG, a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning, conditioned on the target dataset. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation, learning the weight distributions of models pretrained on various datasets. This allows for automatic generation of weights that generalize well across both seen and unseen tasks, outperforming state-of-the-art meta-learning methods and pretrained models. Moreover, our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques that rely on task-specific model collections or access to original training data. By modeling the parameter distribution of LLMs, D2NWG enables task-specific parameter generation without requiring additional fine-tuning or large collections of model variants. Extensive experiments show that our method consistently enhances the performance of diverse base models, regardless of their size or complexity, positioning it as a robust solution for scalable transfer learning.