Maximizing Efficiency of Language Model Pre-training for Learning Representation

TL;DR

This paper proposes an adaptive early exit strategy to enhance the compute efficiency of ELECTRA pre-training by utilizing earlier layer representations, and investigates the necessity of the generator module for maintaining accuracy.

Contribution

It introduces an adaptive early exit method for ELECTRA pre-training and evaluates the generator module's role in balancing efficiency and accuracy.

Findings

Early exit improves training efficiency without significant accuracy loss.

The generator module's necessity is context-dependent and can be reduced.

The approach achieves faster pre-training with comparable performance.

Abstract

Pre-trained language models in the past years have shown exponential growth in model parameters and compute time. ELECTRA is a novel approach for improving the compute efficiency of pre-trained language models (e.g. BERT) based on masked language modeling (MLM) by addressing the sample inefficiency problem with the replaced token detection (RTD) task. Our work proposes adaptive early exit strategy to maximize the efficiency of the pre-training process by relieving the model's subsequent layers of the need to process latent features by leveraging earlier layer representations. Moreover, we evaluate an initial approach to the problem that has not succeeded in maintaining the accuracy of the model while showing a promising compute efficiency by thoroughly investigating the necessity of the generator module of ELECTRA.