On the Effectiveness of Incremental Training of Large Language Models

TL;DR

This study evaluates incremental layer-wise training for large language models, finding it less efficient overall than traditional training despite initial computational benefits.

Contribution

It provides a comprehensive analysis of incremental training's effectiveness, revealing its limitations and impact on training efficiency for large language models.

Findings

Incremental training initially reduces computational costs.

Eventually, it requires more total computation to match traditional training.

Incremental approach extends training time significantly.

Abstract

Training large language models is a computationally intensive process that often requires substantial resources to achieve state-of-the-art results. Incremental layer-wise training has been proposed as a potential strategy to optimize the training process by progressively introducing layers, with the expectation that this approach would lead to faster convergence and more efficient use of computational resources. In this paper, we investigate the effectiveness of incremental training for LLMs, dividing the training process into multiple stages where layers are added progressively. Our experimental results indicate that while the incremental approach initially demonstrates some computational efficiency, it ultimately requires greater overall computational costs to reach comparable performance to traditional full-scale training. Although the incremental training process can eventually close the performance gap with the baseline, it does so only after significantly extended continual training. These findings suggest that incremental layer-wise training may not be a viable alternative for training large language models, highlighting its limitations and providing valuable insights into the inefficiencies of this approach.