Optimizing Traversing and Retrieval Speed of Large Breached Databases

TL;DR

This paper presents methods to optimize the traversal and retrieval speed of large breached databases, enabling cost-effective analysis on personal computers for security researchers.

Contribution

It introduces novel techniques to improve traversal efficiency of large CSV-based breached databases without relying on expensive cloud infrastructure.

Findings

Traversal speed improved significantly

Reduced computational resource requirements

Facilitates analysis on personal computers

Abstract

Breached data refers to the unauthorized access, theft, or exposure of confidential or sensitive information. Breaches typically occur when malicious actors or unauthorized users breach secure systems or networks, resulting in compromised personally identifiable information (PII), protected or personal health information (PHI), payment card industry (PCI) information, or other sensitive data. Data breaches are often the result of malicious activities such as hacking, phishing, insider threats, malware, or physical theft. The misuse of breached data can lead to identity theft, fraud, spamming, or blackmailing. Organizations that experience data breaches may face legal and financial consequences, reputational damage, and harm to their customers or users. Breached records are commonly sold on the dark web or made available on various public forums. To counteract these malicious activities, it is possible to collect breached databases and mitigate potential harm. These databases can be quite large, reaching sizes of up to 150 GB or more. Typically, breached data is stored in the CSV (Comma Separated Value) format due to its simplicity and lightweight nature, which reduces storage requirements. Analyzing and traversing large breached databases necessitates substantial computational power. However, this research explores techniques to optimize database traversal speed without the need to rent expensive cloud machines or virtual private servers (VPS). This optimization will enable individual security researchers to analyze and process large databases on their personal computer systems while significantly reducing costs.