ProQA: Structural Prompt-based Pre-training for Unified Question Answering

TL;DR

ProQA introduces a unified question answering model that uses structural prompts and pre-training on synthesized data to improve generalization across diverse QA tasks, achieving strong results in various learning scenarios.

Contribution

ProQA is the first unified QA framework leveraging structural prompt-based pre-training to handle multiple QA tasks with a single model.

Findings

Consistently improves performance on 11 QA benchmarks.

Effective in full data, few-shot, and zero-shot scenarios.

Shows strong continual and transfer learning capabilities.

Abstract

Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.