TraceScope: Interactive URL Triage via Decoupled Checklist Adjudication

TL;DR

TraceScope is an interactive URL triage system that uses sandboxed browsing and LLM-based adjudication to detect sophisticated phishing URLs with high precision and recall.

Contribution

The paper introduces TraceScope, a scalable, safe, and evidence-based URL triage pipeline that improves detection of evasive phishing URLs using decoupled analysis and LLMs.

Findings

Achieves 0.94 precision and 0.78 recall on test URLs.

Outperforms prior classifiers in recall and detection of evasive phishing.

Successfully detects sophisticated phishing in real-world email datasets.

Abstract

Modern phishing campaigns increasingly evade snapshot-based URL classifiers using interaction gates (e.g., checkbox/slider challenges), delayed content rendering, and logo-less credential harvesters. This shifts URL triage from static classification toward an interactive forensics task: an analyst must actively navigate the page while isolating themselves from potential runtime exploits. We present TraceScope, a decoupled triage pipeline that operationalizes this workflow at scale. To prevent the observer effect and ensure safety, a sandboxed operator agent drives a real GUI browser guided by visual motivation to elicit page behavior, freezing the session into an immutable evidence bundle. Separately, an adjudicator agent circumvents LLM context limitations by querying evidence on demand to verify a MITRE ATT&CK checklist, and generates an audit-ready report with extracted indicators of compromise (IOCs) and a final verdict. Evaluated on 708 reachable URLs from existing dataset (241 verified phishing from PhishTank and 467 benign from Tranco-derived crawling), TraceScope achieves 0.94 precision and 0.78 recall, substantially improving recall over three prior visual/reference-based classifiers while producing reproducible, analyst-grade evidence suitable for review. More importantly, we manually curated a dataset of real-world phishing emails to evaluate our system in a practical setting. Our evaluation reveals that TraceScope demonstrates superior performance in a real-world scenario as well, successfully detecting sophisticated phishing attempts that current state-of-the-art defenses fail to identify.