FoundationLayerNorm: Scaling BERT and GPT to 1,000 Layers

TL;DR

This paper introduces FoundationLayerNormalization, a method that enables training of extremely deep BERT and GPT models with up to 1,000 layers, significantly deeper than previous models, by improving training stability.

Contribution

The paper presents a novel normalization technique that allows stable training of BERT and GPT models with 1,000 layers, surpassing previous depth limitations.

Findings

Successfully trained 1,000-layer BERT and GPT models

FoundationLayerNormalization improves training stability for deep networks

Demonstrates scalability of transformer models to unprecedented depths

Abstract

The mainstream BERT/GPT model contains only 10 to 20 layers, and there is little literature to discuss the training of deep BERT/GPT. This paper proposes a simple yet effective method to stabilize BERT and GPT training. We successfully scale up BERT and GPT to 1,000 layers, which is an order of magnitude deeper than previous BERT and GPT. The proposed method FoundationLayerNormalization enables efficient training of deep neural networks and is validated at the 1000-layer scale.