Large Multimodal Agents for Accurate Phishing Detection with Enhanced Token Optimization and Cost Reduction

TL;DR

This paper demonstrates that large multimodal AI agents can effectively detect phishing websites by analyzing URLs and screenshots, and introduces a cost-efficient, two-tiered agentic approach that significantly reduces API expenses while maintaining high detection accuracy.

Contribution

The paper presents a novel two-tiered agentic method that reduces API costs in multimodal phishing detection without sacrificing performance.

Findings

Integrating URLs and screenshots improves detection accuracy.

The agentic approach reduces API costs by up to 4.2 times.

Cost-effective detection enables scalable phishing defense.

Abstract

With the rise of sophisticated phishing attacks, there is a growing need for effective and economical detection solutions. This paper explores the use of large multimodal agents, specifically Gemini 1.5 Flash and GPT-4o mini, to analyze both URLs and webpage screenshots via APIs, thus avoiding the complexities of training and maintaining AI systems. Our findings indicate that integrating these two data types substantially enhances detection performance over using either type alone. However, API usage incurs costs per query that depend on the number of input and output tokens. To address this, we propose a two-tiered agentic approach: initially, one agent assesses the URL, and if inconclusive, a second agent evaluates both the URL and the screenshot. This method not only maintains robust detection performance but also significantly reduces API costs by minimizing unnecessary multi-input queries. Cost analysis shows that with the agentic approach, GPT-4o mini can process about 4.2 times as many websites per $100 compared to the multimodal approach (107,440 vs. 25,626), and Gemini 1.5 Flash can process about 2.6 times more websites (2,232,142 vs. 862,068). These findings underscore the significant economic benefits of the agentic approach over the multimodal method, providing a viable solution for organizations aiming to leverage advanced AI for phishing detection while controlling expenses.