Plug-and-Play Adaptation for Continuously-updated QA

TL;DR

This paper introduces a new task, CuQA, to evaluate language models' ability to handle multiple large-scale knowledge updates, and proposes plug-in modules that outperform fine-tuning in maintaining and updating knowledge.

Contribution

The paper proposes the CuQA task for continuous knowledge updates and introduces plug-in modules that improve update efficiency and knowledge retention in language models.

Findings

Our method is 4x more effective than fine-tuning in update/forget ratio.

Plug-in modules outperform existing approaches on QA datasets.

The approach effectively balances knowledge updating and retention.

Abstract

Language models (LMs) have shown great potential as implicit knowledge bases (KBs). And for their practical use, knowledge in LMs need to be updated periodically. However, existing tasks to assess LMs' efficacy as KBs do not adequately consider multiple large-scale updates. To this end, we first propose a novel task--Continuously-updated QA (CuQA)--in which multiple large-scale updates are made to LMs, and the performance is measured with respect to the success in adding and updating knowledge while retaining existing knowledge. We then present LMs with plug-in modules that effectively handle the updates. Experiments conducted on zsRE QA and NQ datasets show that our method outperforms existing approaches. We find that our method is 4x more effective in terms of updates/forgets ratio, compared to a fine-tuning baseline.