Reminding the Incremental Language Model via Data-Free Self-Distillation

TL;DR

This paper introduces a data-free self-distillation method for incremental language learning that reduces pseudo-data requirements and mitigates catastrophic forgetting by estimating knowledge distribution and augmenting hidden data.

Contribution

The proposed DFSD method leverages Earth Mover's Distance and hidden data augmentation to improve incremental language learning without extensive pseudo-data.

Findings

Outperforms previous methods with up to 90% less pseudo-data

Significantly reduces catastrophic forgetting in incremental learning

Effective in maintaining knowledge across multiple tasks

Abstract

Incremental language learning with pseudo-data can alleviate catastrophic forgetting in neural networks. However, to obtain better performance, former methods have higher demands for pseudo-data of the previous tasks. The performance dramatically decreases when fewer pseudo-data are employed. In addition, the distribution of pseudo-data gradually deviates from the real data with the sequential learning of different tasks. The deviation will be greater with more tasks learned, which results in more serious catastrophic forgetting. To address these issues, we propose reminding incremental language model via data-free self-distillation (DFSD), which includes self-distillation based on the Earth Mover's Distance and hidden data augmentation. By estimating the knowledge distribution in all layers of GPT-2 and transforming it from teacher model to student model, the Self-distillation based on the Earth Mover's Distance can significantly reduce the demand for pseudo-data. Hidden data augmentation can greatly alleviate the catastrophic forgetting caused by deviations via modeling the generation of pseudo-data as a hidden data augmentation process, where each sample is a mixture of all trained task data. The experimental results demonstrate that our DFSD can exceed the previous state-of-the-art methods even if the maximum decrease in pseudo-data is 90%.