Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts

TL;DR

This study evaluates 15 large language models' ability to detect phishing emails, finding that ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT are most effective in identifying malicious messages.

Contribution

The paper provides an empirical comparison of multiple LLMs' effectiveness in phishing detection, highlighting the top-performing models for cybersecurity applications.

Findings

ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT are most effective

LLMs can accurately detect phishing emails based on email metadata

The study offers insights into LLMs' potential in cybersecurity defenses

Abstract

Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. By leveraging clever social engineering elements and modern technology, cybercrime targets many individuals, businesses, and organizations to exploit trust and security. These cyber-attackers are often disguised in many trustworthy forms to appear as legitimate sources. By cleverly using psychological elements like urgency, fear, social proof, and other manipulative strategies, phishers can lure individuals into revealing sensitive and personalized information. Building on this pervasive issue within modern technology, this paper aims to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts, specifically focusing on a randomized set of "419 Scam" emails. The objective is to determine which LLMs can accurately detect phishing emails by analyzing a text file containing email metadata based on predefined criteria. The experiment concluded that the following models, ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT, were the most effective in detecting phishing emails.