User-Centric Phishing Detection: A RAG and LLM-Based Approach

TL;DR

This paper introduces a personalized phishing detection system combining large language models with retrieval-augmented generation, significantly reducing false positives and improving detection accuracy by leveraging user-specific email history and threat intelligence.

Contribution

It presents a novel RAG and LLM-based framework for personalized phishing detection that outperforms traditional methods in reducing false positives and adapting to individual user patterns.

Findings

Llama4-Scout achieves an F1-score of 0.9703.

RAG reduces false positives by 66.7%.

The approach is effective and adaptable for high-precision email security.

Abstract

The escalating sophistication of phishing emails necessitates a shift beyond traditional rule-based and conventional machine-learning-based detectors. Although large language models (LLMs) offer strong natural language understanding, using them as standalone classifiers often yields elevated falsepositive (FP) rates, which mislabel legitimate emails as phishing and create significant operational burden. This paper presents a personalized phishing detection framework that integrates LLMs with retrieval-augmented generation (RAG). For each message, the system constructs user-specific context by retrieving a compact set of the user's historical legitimate emails and enriching it with real-time domain and URL reputation from a cyber-threat intelligence platform, then conditions the LLM's decision on this evidence. We evaluate four open-source LLMs (Llama4-Scout, DeepSeek-R1, Mistral-Saba, and Gemma2) on an email dataset collected from public and institutional sources. Results show high performance; for example, Llama4-Scout attains an F1-score of 0.9703 and achieves a 66.7% reduction in FPs with RAG. These findings validate that a RAG-based, user-profiling approach is both feasible and effective for building high-precision, low-friction email security systems that adapt to individual communication patterns.