Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research

TL;DR

This paper identifies three types of LLM-related threats—partial mediation, full delegation, and spillover—that undermine the validity of online behavioral research, and proposes multi-layered mitigation strategies.

Contribution

It introduces a comprehensive framework for understanding LLM Pollution variants and discusses strategies for safeguarding research integrity against AI-driven distortions.

Findings

LLM Pollution variants distort research data and interpretations.

Anticipated LLM presence influences participant behavior.

Mitigation requires coordinated practices across researchers, platforms, and communities.

Abstract

Online behavioural research faces an emerging threat as participants increasingly turn to large language models (LLMs) for advice, translation, or task delegation: LLM Pollution. We identify three interacting variants through which LLM Pollution threatens the validity and integrity of online behavioural research. First, Partial LLM Mediation occurs when participants make selective use of LLMs for specific aspects of a task, such as translation or wording support, leading researchers to (mis)interpret LLM-shaped outputs as human ones. Second, Full LLM Delegation arises when agentic LLMs complete studies with little to no human oversight, undermining the central premise of human-subject research at a more foundational level. Third, LLM Spillover signifies human participants altering their behaviour as they begin to anticipate LLM presence in online studies, even when none are involved. While Partial Mediation and Full Delegation form a continuum of increasing automation, LLM Spillover reflects second-order reactivity effects. Together, these variants interact and generate cascading distortions that compromise sample authenticity, introduce biases that are difficult to detect post hoc, and ultimately undermine the epistemic grounding of online research on human cognition and behaviour. Crucially, the threat of LLM Pollution is already co-evolving with advances in generative AI, creating an escalating methodological arms race. To address this, we propose a multi-layered response spanning researcher practices, platform accountability, and community efforts. As the challenge evolves, coordinated adaptation will be essential to safeguard methodological integrity and preserve the validity of online behavioural research.