Self-Calibrating Neural-Probabilistic Model for Authorship Verification Under Covariate Shift

TL;DR

This paper introduces a self-calibrating neural-probabilistic model for authorship verification that effectively handles topic variability and miscalibration, improving accuracy and probability estimates in authorship verification tasks.

Contribution

It extends previous AV frameworks by incorporating Bayes factor scoring and an uncertainty adaptation layer to address topic variability and calibration issues.

Findings

Significantly reduces sensitivity to topical variations.

Improves calibration of probability estimates.

Enhances authorship verification accuracy.

Abstract

We are addressing two fundamental problems in authorship verification (AV): Topic variability and miscalibration. Variations in the topic of two disputed texts are a major cause of error for most AV systems. In addition, it is observed that the underlying probability estimates produced by deep learning AV mechanisms oftentimes do not match the actual case counts in the respective training data. As such, probability estimates are poorly calibrated. We are expanding our framework from PAN 2020 to include Bayes factor scoring (BFS) and an uncertainty adaptation layer (UAL) to address both problems. Experiments with the 2020/21 PAN AV shared task data show that the proposed method significantly reduces sensitivities to topical variations and significantly improves the system's calibration.