Towards Ethical Content-Based Detection of Online Influence Campaigns

TL;DR

This paper explores a text-based method for detecting online influence campaigns, introducing named entity masking to improve robustness and discussing ethical implications of language biases.

Contribution

It proposes named entity masking to reduce false positives in influence detection and highlights ethical challenges related to language biases.

Findings

Named entity masking reduces false positives when key entities are mentioned.

Both masked and unmasked models show increased false positives with Russian native speakers.

Ethical considerations regarding language bias are identified for future research.

Abstract

The detection of clandestine efforts to influence users in online communities is a challenging problem with significant active development. We demonstrate that features derived from the text of user comments are useful for identifying suspect activity, but lead to increased erroneous identifications when keywords over-represented in past influence campaigns are present. Drawing on research in native language identification (NLI), we use "named entity masking" (NEM) to create sentence features robust to this shortcoming, while maintaining comparable classification accuracy. We demonstrate that while NEM consistently reduces false positives when key named entities are mentioned, both masked and unmasked models exhibit increased false positive rates on English sentences by Russian native speakers, raising ethical considerations that should be addressed in future research.