Task Transfer and Domain Adaptation for Zero-Shot Question Answering

TL;DR

This paper explores how combining task transfer and domain adaptation techniques enables pretrained language models to perform zero-shot question answering in new domains without labeled data, improving over existing methods.

Contribution

It introduces a method that effectively combines task transfer with domain adaptation for zero-shot question answering, outperforming previous approaches in most tested domains.

Findings

Outperforms Domain-Adaptive Pretraining in 3 out of 4 domains

Enables zero-shot domain-specific reading comprehension

Reduces need for labeled data in new domains

Abstract

Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domain-specific downstream tasks. We evaluate zero-shot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms Domain-Adaptive Pretraining on downstream domain-specific reading comprehension tasks in 3 out of 4 domains.