Synthetic Question Value Estimation for Domain Adaptation of Question Answering

TL;DR

This paper introduces a question value estimator (QVE) that directly predicts the usefulness of synthetic questions for improving domain-specific QA, leading to better performance with less human annotation.

Contribution

The novel QVE approach directly estimates question usefulness for domain adaptation, outperforming existing scoring methods in QA performance.

Findings

QVE-selected questions improve target-domain QA accuracy

Achieves comparable results with only 15% of human annotations

Outperforms existing question scoring techniques

Abstract

Synthesizing QA pairs with a question generator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models. Since synthetic questions are often noisy in practice, existing work adapts scores from a pretrained QA (or QG) model as criteria to select high-quality questions. However, these scores do not directly serve the ultimate goal of improving QA performance on the target domain. In this paper, we introduce a novel idea of training a question value estimator (QVE) that directly estimates the usefulness of synthetic questions for improving the target-domain QA performance. By conducting comprehensive experiments, we show that the synthetic questions selected by QVE can help achieve better target-domain QA performance, in comparison with existing techniques. We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable performance to the fully-supervised baselines.