Continuous QA Learning with Structured Prompts

TL;DR

This paper introduces Diana, a dynamic prompt-enhanced language model for lifelong question answering that effectively learns from sequential tasks, including unseen ones, by using hierarchically organized prompts for knowledge retention and generalization.

Contribution

Diana is a novel architecture that employs multiple levels of prompts to improve lifelong QA performance and handle unseen tasks without requiring task identities during testing.

Findings

Diana outperforms existing lifelong QA models in various benchmarks.

Hierarchical prompts improve knowledge retention and generalization.

Explicit modeling of unseen tasks enhances performance on new questions.

Abstract

QA models with lifelong learning (LL) abilities are important for practical QA applications, and architecture-based LL methods are reported to be an effective implementation for these models. However, it is non-trivial to extend previous approaches to QA tasks since they either require access to task identities in the testing phase or do not explicitly model samples from unseen tasks. In this paper, we propose Diana: a dynamic architecture-based lifelong QA model that tries to learn a sequence of QA tasks with a prompt enhanced language model. Four types of hierarchically organized prompts are used in Diana to capture QA knowledge from different granularities. Specifically, we dedicate task-level prompts to capture task-specific knowledge to retain high LL performances and maintain instance-level prompts to learn knowledge shared across different input samples to improve the model's generalization performance. Moreover, we dedicate separate prompts to explicitly model unseen tasks and introduce a set of prompt key vectors to facilitate knowledge sharing between tasks. Extensive experiments demonstrate that Diana outperforms state-of-the-art lifelong QA models, especially in handling unseen tasks.