Making Sense of Scams: Understanding Scam Conversations Through Multi-Level Alignment

TL;DR

This paper introduces multi-level alignment-based hints derived from conversational dynamics to improve scam detection, demonstrating enhanced accuracy and earlier confidence in identifying scam conversations.

Contribution

It proposes a novel multi-level alignment approach based on the Interactive Alignment Model to support non-interruptive scam detection in conversations.

Findings

Alignment scores decline as scams approach, indicating scam progression.

Alignment-based hints improve detection precision by 0.25 and recall by 0.16.

Hints support earlier, more stable confidence formation in scam detection.

Abstract

Online scams often unfold gradually through interaction, yet existing detection systems predominantly rely on snapshot-based signals and interruptive warnings, revealing two research gaps in the lack of signals that represent scam risk within conversational dynamics and the underexplored design of non-interruptive interaction. To address these gaps, we introduce multi-level alignment-based hints, informed by the Interactive Alignment Model, as a new detection signal for supporting sensemaking in scam-related conversations. These hints operationalize low-level lexical and syntactic alignments and high-level semantic and situation-model alignments between conversational participants, making conversational dynamics visible to users. We first conduct a preliminary evaluation on real-life scam dialogues, showing that as conversations approach scam attempts, low-level alignment scores remain stable while high-level alignment scores systematically decline, revealing a consistent cross-level pattern indicative of scam progression. Building on this insight, we conduct a user study with thirty participants, indicating that relative to the no-hint baseline, multi-level alignment-based hints increase precision by 0.25, recall by 0.16, and F1 score by 0.21, yielding substantially larger gains than the marginal improvements achieved by keyword-triggered alerts. Statistical analyses reveal that the proposed hints support earlier and more stable confidence formation over time, with ablation results further highlighting the effectiveness of combining alignment hints across levels in achieving these advantages.