Deep Fusion: Efficient Network Training via Pre-trained Initializations

TL;DR

Deep Fusion introduces an efficient training method for deep neural networks using pre-trained small networks and a theoretical framework to optimize training dynamics, significantly reducing training time and computational resources in NLP tasks.

Contribution

The paper presents a novel network training approach called Deep Fusion and a theoretical framework for understanding and optimizing network growth during training.

Findings

Deep Fusion accelerates training and reduces computational costs.

It maintains or improves performance compared to traditional methods.

The theoretical framework guides optimal training dynamics.

Abstract

In recent years, deep learning has made remarkable progress in a wide range of domains, with a particularly notable impact on natural language processing tasks. One of the challenges associated with training deep neural networks in the context of LLMs is the need for large amounts of computational resources and time. To mitigate this, network growing algorithms offer potential cost savings, but their underlying mechanisms are poorly understood. We present two notable contributions in this paper. First, we present Deep Fusion, an efficient approach to network training that leverages pre-trained initializations of smaller networks. Second, we propose a theoretical framework using backward error analysis to illustrate the dynamics of mid-training network growth. Our experiments show how Deep Fusion is a practical and effective approach that not only accelerates the training process but also reduces computational requirements, maintaining or surpassing traditional training methods' performance in various NLP tasks and T5 model sizes. Finally, we validate our theoretical framework, which guides the optimal use of Deep Fusion, showing that with carefully optimized training dynamics, it significantly reduces both training time and resource consumption.