The Topic Confusion Task: A Novel Scenario for Authorship Attribution

TL;DR

The paper introduces the topic confusion task for authorship attribution, which distinguishes errors caused by topic shifts from those caused by stylistic features, revealing the robustness of stylometric features and limitations of pretrained language models.

Contribution

It proposes a novel topic confusion scenario for authorship attribution to analyze error sources and evaluates stylometric features versus pretrained models under topic shifts.

Findings

Stylometric features with POS tags are less affected by topic changes.

Combining features reduces topic confusion and improves accuracy.

Pretrained models like BERT and RoBERTa perform poorly compared to simple n-gram features.

Abstract

Authorship attribution is the problem of identifying the most plausible author of an anonymous text from a set of candidate authors. Researchers have investigated same-topic and cross-topic scenarios of authorship attribution, which differ according to whether new, unseen topics are used in the testing phase. However, neither scenario allows us to explain whether errors are caused by a failure to capture authorship writing style or by a topic shift. Motivated by this, we propose the \emph{topic confusion} task where we switch the author-topic configuration between the training and testing sets. This setup allows us to distinguish two types of errors: those caused by the topic shift and those caused by the features' inability to capture the writing styles. We show that stylometric features with part-of-speech tags are the least susceptible to topic variations. We further show that combining them with other features leads to significantly lower topic confusion and higher attribution accuracy. Finally, we show that pretrained language models such as BERT and RoBERTa perform poorly on this task and are surpassed by simple features such as word-level $n$-grams.