In-context learning for the classification of manipulation techniques in phishing emails

TL;DR

This paper explores using Large Language Model In-Context Learning to classify phishing emails based on manipulation techniques, achieving promising accuracy and offering insights into attacker strategies.

Contribution

It introduces a novel application of LLM ICL for fine-grained phishing email classification using a detailed taxonomy of manipulation techniques.

Findings

Achieved 76% accuracy in identifying manipulation techniques.

Effectively distinguished prevalent techniques like Baiting and Curiosity Appeal.

Demonstrated ICL's potential for nuanced phishing analysis.

Abstract

Traditional phishing detection often overlooks psychological manipulation. This study investigates using Large Language Model (LLM) In-Context Learning (ICL) for fine-grained classification of phishing emails based on a taxonomy of 40 manipulation techniques. Using few-shot examples with GPT-4o-mini on real-world French phishing emails (SignalSpam), we evaluated performance against a human-annotated test set (100 emails). The approach effectively identifies prevalent techniques (e.g., Baiting, Curiosity Appeal, Request For Minor Favor) with a promising accuracy of 0.76. This work demonstrates ICL's potential for nuanced phishing analysis and provides insights into attacker strategies.