Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering

TL;DR

This paper empirically investigates domain generalization in question answering, revealing that better learning of source domains, rather than overfitting, enhances zero-shot out-of-domain performance, challenging common assumptions.

Contribution

It demonstrates that improving source domain learning via knowledge distillation enhances out-of-domain generalization in question answering, shifting focus from overfitting to underfitting mitigation.

Findings

Knowledge distillation improves zero-shot out-of-domain performance.

Better source domain learning correlates with improved generalization.

Existing DG methods focusing on limiting overfitting are less effective.

Abstract

Machine learning models are prone to overfitting their training (source) domains, which is commonly believed to be the reason why they falter in novel target domains. Here we examine the contrasting view that multi-source domain generalization (DG) is first and foremost a problem of mitigating source domain underfitting: models not adequately learning the signal already present in their multi-domain training data. Experiments on a reading comprehension DG benchmark show that as a model learns its source domains better -- using familiar methods such as knowledge distillation (KD) from a bigger model -- its zero-shot out-of-domain utility improves at an even faster pace. Improved source domain learning also demonstrates superior out-of-domain generalization over three popular existing DG approaches that aim to limit overfitting. Our implementation of KD-based domain generalization is available via PrimeQA at: https://ibm.biz/domain-generalization-with-kd.