Depth $F_1$: Improving Evaluation of Cross-Domain Text Classification by Measuring Semantic Generalizability

TL;DR

Depth F1 is a new metric for cross-domain text classification that evaluates how well models perform on target samples dissimilar from the source, addressing limitations of existing evaluation methods.

Contribution

The paper introduces Depth F1, a novel metric that measures semantic generalizability by focusing on dissimilar target samples in cross-domain classification.

Findings

Depth F1 effectively highlights models' performance on dissimilar target samples.

Benchmarking shows varying model capabilities in semantic transfer.

Depth F1 complements existing metrics by providing deeper evaluation insights.

Abstract

Recent evaluations of cross-domain text classification models aim to measure the ability of a model to obtain domain-invariant performance in a target domain given labeled samples in a source domain. The primary strategy for this evaluation relies on assumed differences between source domain samples and target domain samples in benchmark datasets. This evaluation strategy fails to account for the similarity between source and target domains, and may mask when models fail to transfer learning to specific target samples which are highly dissimilar from the source domain. We introduce Depth $F_1$, a novel cross-domain text classification performance metric. Designed to be complementary to existing classification metrics such as $F_1$, Depth $F_1$ measures how well a model performs on target samples which are dissimilar from the source domain. We motivate this metric using standard cross-domain text classification datasets and benchmark several recent cross-domain text classification models, with the goal of enabling in-depth evaluation of the semantic generalizability of cross-domain text classification models.