SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors

TL;DR

The paper introduces SEAR, a comprehensive multimodal dataset capturing AR-driven social engineering interactions, revealing high attack success rates and supporting research on detection and defense against such threats.

Contribution

It provides the first multimodal dataset specifically designed for studying AR-LLM-driven social engineering behaviors, including annotated multimodal cues and trust metrics.

Findings

High success rate of social engineering attacks in simulated scenarios

Dataset enables research on detection and mitigation of AR-driven SE attacks

Supports understanding of multimodal manipulation in social engineering

Abstract

The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.