PreScam: A Benchmark for Predicting Scam Progression from Early Conversations

TL;DR

PreScam is a comprehensive benchmark dataset designed to evaluate how well language models can understand and predict the progression of online conversational scams over multiple turns.

Contribution

The paper introduces PreScam, a large structured dataset of scam conversations with annotations, and benchmarks models on tasks predicting scam progression and actions.

Findings

Supervised encoders outperform zero-shot LLMs in termination prediction.

Next-action prediction is only moderately successful even for strong LLMs.

Current models struggle to fully capture scam escalation and manipulation cues.

Abstract

Conversational scams, such as romance and investment scams, are emerging as a major form of online fraud. Unlike one-shot scam lures such as fake lottery or unpaid toll messages, they unfold through multi-turn conversations in which scammers gradually manipulate victims using evolving psychological techniques. However, existing research mainly focuses on static scam detection or synthetic scams, leaving open whether language models can understand how real-world scams progress over time. We introduce PreScam, a benchmark for modeling scam progression from early conversations. Built from user-submitted scam reports, PreScam filters and structures 177,989 raw reports into 11,573 conversational scam instances spanning 20 scam categories. Each instance is hierarchically structured according to the scam lifecycle defined by the proposed scam kill chain, and further annotated at the turn level with scammer psychological actions and victim responses. We benchmark models on two tasks: real-time termination prediction, which estimates whether a conversation is approaching the termination stage, and scammer action prediction, which forecasts the scammer's subsequent actions. Results show a clear gap between surface-level fluency and progression modeling: supervised encoders substantially outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs. Taken together, these results show that current models can capture some scam-related cues, yet still struggle to track how risk escalates and how manipulation unfolds across turns.