Extending the Scope of Out-of-Domain: Examining QA models in multiple subdomains

TL;DR

This paper investigates how question answering models perform across various subdomains defined by internal dataset characteristics, revealing significant performance drops when training and testing data differ in these subdomains.

Contribution

It introduces a new perspective by analyzing QA model performance across subdomains based on internal dataset features, highlighting the limitations of current generalization assumptions.

Findings

Performance drops when train and test data are from different subdomains.

Internal dataset characteristics significantly impact QA system accuracy.

Current QA models may not generalize well across diverse subdomains.

Abstract

Past works that investigate out-of-domain performance of QA systems have mainly focused on general domains (e.g. news domain, wikipedia domain), underestimating the importance of subdomains defined by the internal characteristics of QA datasets. In this paper, we extend the scope of "out-of-domain" by splitting QA examples into different subdomains according to their several internal characteristics including question type, text length, answer position. We then examine the performance of QA systems trained on the data from different subdomains. Experimental results show that the performance of QA systems can be significantly reduced when the train data and test data come from different subdomains. These results question the generalizability of current QA systems in multiple subdomains, suggesting the need to combat the bias introduced by the internal characteristics of QA datasets.