LLaMA Pro: Progressive LLaMA with Block Expansion

TL;DR

LLaMA Pro introduces a progressive expansion method for LLMs by adding Transformer blocks and fine-tuning with new data, enhancing knowledge without forgetting, and achieving superior performance across tasks.

Contribution

The paper presents a novel post-pretraining block expansion technique for LLMs, improving knowledge retention and task versatility without catastrophic forgetting.

Findings

LLaMA Pro-8.3B outperforms existing open models in benchmarks.

Effective knowledge integration from new data without forgetting.

Demonstrates strong performance in programming and mathematics tasks.

Abstract

Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.