Switching Contexts: Transportability Measures for NLP

TL;DR

This paper introduces transportability metrics based on established statistics to estimate NLP model performance changes across different domains and tasks, aiding in better generalization assessment.

Contribution

It proposes new lightweight domain similarity measures for estimating NLP system performance in new contexts, enhancing transportability evaluation methods.

Findings

Transportability measures effectively predict performance drops in NER and NLI tasks.

Lightweight domain similarity metrics serve as reliable estimators for NLP model transferability.

The approach improves understanding of model robustness across diverse domains.

Abstract

This paper explores the topic of transportability, as a sub-area of generalisability. By proposing the utilisation of metrics based on well-established statistics, we are able to estimate the change in performance of NLP models in new contexts. Defining a new measure for transportability may allow for better estimation of NLP system performance in new domains, and is crucial when assessing the performance of NLP systems in new tasks and domains. Through several instances of increasing complexity, we demonstrate how lightweight domain similarity measures can be used as estimators for the transportability in NLP applications. The proposed transportability measures are evaluated in the context of Named Entity Recognition and Natural Language Inference tasks.