Addressing Topic Leakage in Cross-Topic Evaluation for Authorship Verification

TL;DR

This paper introduces HITS, a new evaluation method for authorship verification that reduces topic leakage effects, leading to more stable model rankings and revealing models' reliance on topic-specific cues.

Contribution

It proposes HITS for better evaluation of AV models under topic shift and introduces RAVEN benchmark to test topic shortcut reliance.

Findings

HITS creates more stable model rankings across different splits.

RAVEN uncovers models' dependence on topic-specific features.

Topic leakage significantly impacts AV model evaluation.

Abstract

Authorship verification (AV) aims to identify whether a pair of texts has the same author. We address the challenge of evaluating AV models' robustness against topic shifts. The conventional evaluation assumes minimal topic overlap between training and test data. However, we argue that there can still be topic leakage in test data, causing misleading model performance and unstable rankings. To address this, we propose an evaluation method called Heterogeneity-Informed Topic Sampling (HITS), which creates a smaller dataset with a heterogeneously distributed topic set. Our experimental results demonstrate that HITS-sampled datasets yield a more stable ranking of models across random seeds and evaluation splits. Our contributions include: 1. An analysis of causes and effects of topic leakage. 2. A demonstration of the HITS in reducing the effects of topic leakage, and 3. The Robust Authorship Verification bENchmark (RAVEN) that allows topic shortcut test to uncover AV models' reliance on topic-specific features.