Unmasking Social Bots: How Confident Are We?

TL;DR

This paper tackles the challenge of social bot detection by not only developing algorithms to identify bots but also quantifying the uncertainty of each prediction to improve decision-making and reliability.

Contribution

It introduces a novel approach that combines bot detection with uncertainty quantification at the account level, enhancing trust and targeted interventions.

Findings

Detection models often disagree on classifications.

Uncertainty measures can guide intervention strategies.

The approach improves decision confidence in bot detection.

Abstract

Social bots remain a major vector for spreading disinformation on social media and a menace to the public. Despite the progress made in developing multiple sophisticated social bot detection algorithms and tools, bot detection remains a challenging, unsolved problem that is fraught with uncertainty due to the heterogeneity of bot behaviors, training data, and detection algorithms. Detection models often disagree on whether to label the same account as bot or human-controlled. However, they do not provide any measure of uncertainty to indicate how much we should trust their results. We propose to address both bot detection and the quantification of uncertainty at the account level - a novel feature of this research. This dual focus is crucial as it allows us to leverage additional information related to the quantified uncertainty of each prediction, thereby enhancing decision-making and improving the reliability of bot classifications. Specifically, our approach facilitates targeted interventions for bots when predictions are made with high confidence and suggests caution (e.g., gathering more data) when predictions are uncertain.