Tracking Network Events with Write Optimized Data Structures: The Design and Implementation of TWIAD: The Write-Optimized IP Address Database

TL;DR

This paper introduces TWIAD, a write-optimized IP address database using B-epsilon trees, enabling high-speed, long-term network event tracking on modest hardware with 20,000 inserts per second.

Contribution

The paper presents TWIAD, a novel database system leveraging write-optimized data structures for efficient long-term network event storage and analysis.

Findings

Achieved 20,000 inserts per second on a desktop system

Demonstrated feasibility of long-term network event tracking with low-cost hardware

Showed write-optimized data structures significantly improve data ingestion rates

Abstract

Access to network traffic records is an integral part of recognizing and addressing network security breaches. Even with the increasing sophistication of network attacks, basic network events such as connections between two IP addresses play an important role in any network defense. Given the duration of current attacks, long-term data archival is critical but typically very little of the data is ever accessed. Previous work has provided tools and identified the need to trace connections. However, traditional databases raise performance concerns as they are optimized for querying rather than ingestion. The study of write-optimized data structures (WODS) is a new and growing field that provides a novel approach to traditional storage structures (e.g., B-trees). WODS trade minor degradations in query performance for significant gains in the ability to quickly insert more data elements, typically on the order of 10 to 100 times more inserts per second. These efficient, out-of-memory data structures can play a critical role in enabling robust, long-term tracking of network events. In this paper, we present TWIAD, the Write-optimized IP Address Database. TWIAD uses a write-optimized B-tree known as a B {\epsilon} tree to track all IP address connections in a network traffic stream. Our initial implementation focuses on utilizing lower cost hardware, demonstrating that basic long-term tracking can be done without advanced equipment. We tested TWIAD on a modest desktop system and showed a sustained ingestion rate of about 20,000 inserts per second.