THREAT/crawl: a Trainable, Highly-Reusable, and Extensible Automated Method and Tool to Crawl Criminal Underground Forums

TL;DR

THREAT/crawl is a versatile, trainable tool designed to efficiently and discreetly crawl diverse underground criminal forums, adapting to various structures and remaining under the radar for security research and operations.

Contribution

It introduces a highly reusable, extensible crawling method and prototype capable of learning arbitrary forum structures and staying covert during operation.

Findings

Successfully crawled multiple active underground forums

Demonstrated adaptability to diverse forum structures

Maintained low detectability during crawling activities

Abstract

Collecting data on underground criminal communities is highly valuable both for security research and security operations. Unfortunately these communities live within a constellation of diverse online forums that are difficult to infiltrate, may adopt crawling monitoring countermeasures, and require the development of ad-hoc scrapers for each different community, making the endeavour increasingly technically challenging, and potentially expensive. To address this problem we propose THREAT/crawl, a method and prototype tool for a highly reusable crawler that can learn a wide range of (arbitrary) forum structures, can remain under-the-radar during the crawling activity and can be extended and configured at the user will. We showcase THREAT/crawl capabilities and provide prime evaluation of our prototype against a range of active, live, underground communities.