Ask Question First for Enhancing Lifelong Language Learning
Han Wang, Ruiliu Fu, Xuejun Zhang, Jun Zhou, Qingwei Zhao

TL;DR
This paper introduces AQF-RQ, a novel approach for lifelong language learning that improves pseudo data generation and reduces catastrophic forgetting by asking questions first, achieving near multi-task performance.
Contribution
It proposes a new data format and training task that enhance pseudo question generation, improving lifelong learning in NLP with minimal performance loss.
Findings
AQF-RQ generates more accurate pseudo data.
It is robust to varying pseudo-data availability.
Performance is within 0.36% of multi-task learning.
Abstract
Lifelong language learning aims to stream learning NLP tasks while retaining knowledge of previous tasks. Previous works based on the language model and following data-free constraint approaches have explored formatting all data as "begin token (\textit{B}) + context (\textit{C}) + question (\textit{Q}) + answer (\textit{A})" for different tasks. However, they still suffer from catastrophic forgetting and are exacerbated when the previous task's pseudo data is insufficient for the following reasons: (1) The model has difficulty generating task-corresponding pseudo data, and (2) \textit{A} is prone to error when \textit{A} and \textit{C} are separated by \textit{Q} because the information of the \textit{C} is diminished before generating \textit{A}. Therefore, we propose the Ask Question First and Replay Question (AQF-RQ), including a novel data format "\textit{BQCA}" and a new training…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Domain Adaptation and Few-Shot Learning
