ScamFerret: Detecting Scam Websites Autonomously with Large Language Models

TL;DR

ScamFerret uses large language models to autonomously detect scam websites across multiple languages by analyzing web content, DNS, and reviews, achieving high accuracy without additional training.

Contribution

Introduces ScamFerret, an LLM-based agent system that identifies scam websites without training or fine-tuning, leveraging natural language understanding for multilingual detection.

Findings

Achieves 97.2% accuracy in English scam detection

Achieves 99.3% accuracy in multilingual online shopping scams

Effectively analyzes external web data for scam identification

Abstract

With the rise of sophisticated scam websites that exploit human psychological vulnerabilities, distinguishing between legitimate and scam websites has become increasingly challenging. This paper presents ScamFerret, an innovative agent system employing a large language model (LLM) to autonomously collect and analyze data from a given URL to determine whether it is a scam. Unlike traditional machine learning models that require large datasets and feature engineering, ScamFerret leverages LLMs' natural language understanding to accurately identify scam websites of various types and languages without requiring additional training or fine-tuning. Our evaluation demonstrated that ScamFerret achieves 0.972 accuracy in classifying four scam types in English and 0.993 accuracy in classifying online shopping websites across three different languages, particularly when using GPT-4. Furthermore, we confirmed that ScamFerret collects and analyzes external information such as web content, DNS records, and user reviews as necessary, providing a basis for identifying scam websites from multiple perspectives. These results suggest that LLMs have significant potential in enhancing cybersecurity measures against sophisticated scam websites.