Human, AI, and Hybrid Ensembles for Detection of Adaptive, RL-based Social Bots

TL;DR

This study compares humans, AI, and hybrid ensembles in detecting adaptive social bots using reinforcement learning, revealing that combined approaches outperform individual methods and offering insights for cybersecurity practices.

Contribution

First systematic comparison of human, AI, and hybrid detection methods for adaptive RL-based social bots, including new aggregation and retraining strategies.

Findings

Hybrid human-AI detection outperforms individual methods.

Unexpected human detection patterns were observed.

Combining human reports with AI improves detection accuracy.

Abstract

The use of reinforcement learning to dynamically adapt and evade detection is now well-documented in several cybersecurity settings including Covert Social Influence Operations (CSIOs), in which bots try to spread disinformation. While AI bot detectors have improved greatly, they are largely limited to detecting static bots that do not adapt dynamically. We present the first systematic study comparing the ability of humans, AI models, and hybrid Human-AI ensembles in detecting adaptive bots powered by reinforcement learning. Using data from a controlled, IRB-approved, five-day experiment with participants interacting on a social media platform infiltrated by RL-trained bots spreading disinformation to influence participants on 4 topics, we examine factors potentially shaping human detection capabilities: demographic characteristics, temporal learning effects, social network position, engagement patterns, and collective intelligence mechanisms. We first test 13 hypotheses comparing human bot detection performance against state-of-the-art AI approaches utilizing both traditional machine learning and large language models. We further investigate several aggregation strategies that combine human reports of bots with AI predictions, as well as retraining protocols that leverage human supervision. Our findings challenge intuitive assumptions about bot detection, reveal unexpected patterns in how humans identify bots, and show that combining human bot reports with AI predictions outperforms humans alone and AI alone. We conclude with a discussion of the practical implications of these results for industry.