Eliciting Least-to-Most Reasoning for Phishing URL Detection

TL;DR

This paper introduces a Least-to-Most prompting framework with an answer sensitivity mechanism for phishing URL detection using large language models, achieving high accuracy with minimal training data and outperforming baseline methods.

Contribution

The paper presents a novel Least-to-Most prompting approach with answer sensitivity for improved phishing URL detection, demonstrating effectiveness across multiple datasets and models.

Findings

Outperforms one-shot baseline in accuracy

Achieves comparable results to supervised models with less training data

Iterative reasoning enhances detection performance

Abstract

Phishing continues to be one of the most prevalent attack vectors, making accurate classification of phishing URLs essential. Recently, large language models (LLMs) have demonstrated promising results in phishing URL detection. However, their reasoning capabilities that enabled such performance remain underexplored. To this end, in this paper, we propose a Least-to-Most prompting framework for phishing URL detection. In particular, we introduce an "answer sensitivity" mechanism that guides Least-to-Most's iterative approach to enhance reasoning and yield higher prediction accuracy. We evaluate our framework using three URL datasets and four state-of-the-art LLMs, comparing against a one-shot approach and a supervised model. We demonstrate that our framework outperforms the one-shot baseline while achieving performance comparable to that of the supervised model, despite requiring significantly less training data. Furthermore, our in-depth analysis highlights how the iterative reasoning enabled by Least-to-Most, and reinforced by our answer sensitivity mechanism, drives these performance gains. Overall, we show that this simple yet powerful prompting strategy consistently outperforms both one-shot and supervised approaches, despite requiring minimal training or few-shot guidance. Our experimental setup can be found in our Github repository github.sydney.edu.au/htri0928/least-to-most-phishing-detection.