Reproduction and Replication of an Adversarial Stylometry Experiment

TL;DR

This paper reproduces and replicates a key adversarial stylometry experiment, revealing that some defense methods may be less effective than previously thought, especially when considering control groups and automatic translation techniques.

Contribution

It provides a thorough reproduction and replication of a seminal adversarial stylometry study, highlighting potential overestimations of defense effectiveness and the impact of automatic translation.

Findings

Replicated original experiment with consistent results.

Identified potential overstatement of defense effectiveness.

Highlighted the impact of automatic translation on authorship attribution.

Abstract

Maintaining anonymity in natural language communication remains a challenging task. Even when the number of candidate authors is large, standard authorship attribution techniques that analyze writing style predict the original author with uncomfortably high accuracy. Adversarial stylometry provides a defense against authorship attribution, helping users avoid unwanted deanonymization. This paper reproduces and replicates experiments from a seminal study of defenses against authorship attribution (Brennan et al., 2012). After reproducing the experiment using the original data, we then replicate the experiment by repeating the online field experiment using the procedures described in the original paper. Although we reach the same conclusion as the original paper, our results suggest that the defenses studied may be overstated in their effectiveness. This is largely due to the absence of a control group in the original study. In our replication, we find evidence suggesting that an entirely automatic method, round-trip translation, warrants re-examination because it appears to reduce the effectiveness of established authorship attribution methods.