ELLE: Efficient Lifelong Pre-training for Emerging Data

TL;DR

ELLE introduces an efficient lifelong pre-training framework that dynamically expands models and uses domain prompts to adapt to streaming data, improving knowledge integration and downstream task performance.

Contribution

ELLE presents a novel approach combining model expansion and domain prompts for efficient lifelong pre-training on streaming data.

Findings

ELLE outperforms baselines in pre-training efficiency.

ELLE achieves better downstream task performance.

ELLE effectively handles multi-domain streaming data.

Abstract

Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. The codes are publicly available at https://github.com/thunlp/ELLE.