Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams

TL;DR

This study reveals that large language models are already extensively used in romance-baiting scams, outperforming humans in eliciting trust and compliance, while current safety filters fail to detect such scams.

Contribution

The paper provides empirical evidence of LLM deployment in scams, compares LLM and human scam operators, and evaluates safety filter effectiveness against romance-baiting dialogues.

Findings

87% of scam tasks are automatable with LLMs

LLM agents elicited more trust and higher compliance than humans

Safety filters detected 0% of romance baiting conversations

Abstract

Romance-baiting scams have become a major source of financial and emotional harm worldwide. These operations are run by organized crime syndicates that traffic thousands of people into forced labor, requiring them to build emotional intimacy with victims over weeks of text conversations before pressuring them into fraudulent cryptocurrency investments. Because the scams are inherently text-based, they raise urgent questions about the role of Large Language Models (LLMs) in both current and future automation. We investigate this intersection by interviewing 145 insiders and 5 scam victims, performing a blinded long-term conversation study comparing LLM scam agents to human operators, and executing an evaluation of commercial safety filters. Our findings show that LLMs are already widely deployed within scam organizations, with 87% of scam labor consisting of systematized conversational tasks readily susceptible to automation. In a week-long study, an LLM agent not only elicited greater trust from study participants (p=0.007) but also achieved higher compliance with requests than human operators (46% vs. 18% for humans). Meanwhile, popular safety filters detected 0.0% of romance baiting dialogues. Together, these results suggest that romance-baiting scams may be amenable to full-scale LLM automation, while existing defenses remain inadequate to prevent their expansion.