AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

TL;DR

This paper introduces AF Adapter, a continual pretraining method for Chinese biomedical language models that efficiently improves performance and reduces catastrophic forgetting using minimal additional parameters.

Contribution

The paper proposes a novel Attention-FFN Adapter for continual pretraining, effectively enhancing domain-specific Chinese biomedical language models with minimal parameter increase.

Findings

Achieves 0.6% to 2% performance gains over baselines.

Uses only 17% of model parameters for training.

Reduces catastrophic forgetting by 11%.

Abstract

Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method.