@GrokSet: multi-party Human-LLM Interactions in Social Media

TL;DR

This paper introduces @GrokSet, a large dataset of over 1 million tweets involving an LLM on social media, revealing its role as an authority in polarized debates and exposing shallow alignment issues.

Contribution

The paper presents a novel large-scale social media dataset involving an LLM, analyzing its social role, engagement, and safety filter bypass methods in real-world interactions.

Findings

LLM often acts as an authority in political debates

The model receives less social validation than human peers

Users bypass safety filters through simple persona adoption

Abstract

Large Language Models (LLMs) are increasingly deployed as active participants on public social media platforms, yet their behavior in these unconstrained social environments remains largely unstudied. Existing datasets, drawn primarily from private chat interfaces, lack the multi-party dynamics and public visibility crucial for understanding real-world performance. To address this gap, we introduce @GrokSet, a large-scale dataset of over 1 million tweets involving the @Grok LLM on X. Our analysis reveals a distinct functional shift: rather than serving as a general assistant, the LLM is frequently invoked as an authoritative arbiter in high-stakes, polarizing political debates. However, we observe a persistent engagement gap: despite this visibility, the model functions as a low-status utility, receiving significantly less social validation (likes, replies) than human peers. Finally, we find that this adversarial context exposes shallow alignment: users bypass safety filters not through complex jailbreaks, but through simple persona adoption and tone mirroring. We release @GrokSet as a critical resource for studying the intersection of AI agents and societal discourse.