Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme

TL;DR

This paper presents a new neural network initialization method that enhances emergence, leading to improved accuracy and training speed across various architectures without additional optimization steps.

Contribution

The paper introduces a simple, theoretically innovative initialization scheme that promotes emergence, significantly boosting neural network performance and training efficiency.

Findings

Improved model accuracy across multiple architectures

Faster training convergence

Effective without batch normalization

Abstract

Emergence in machine learning refers to the spontaneous appearance of complex behaviors or capabilities that arise from the scale and structure of training data and model architectures, despite not being explicitly programmed. We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence. Measuring emergence as a kind of structural nonlinearity, our method adjusts the layer-wise weight scaling factors to achieve higher emergence values. This enhancement is easy to implement, requiring no additional optimization steps for initialization compared to GradInit. We evaluate our approach across various architectures, including MLP and convolutional architectures for image recognition and transformers for machine translation. We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization. The simplicity, theoretical innovation, and demonstrable empirical advantages of our method make it a potent enhancement to neural network initialization practices. These results suggest a promising direction for leveraging emergence to improve neural network training methodologies. Code is available at: https://github.com/johnnyjingzeli/EmergenceInit.