Tracking Network Events with Write Optimized Data Structures: The Design and Implementation of TWIAD: The Write-Optimized IP Address Database
Nolan Donoghue, Bridger Hahn, Helen Xu, Thomas Kroeger, David Zage and, Rob Johnson

TL;DR
This paper introduces TWIAD, a write-optimized IP address database using B-epsilon trees, enabling high-speed, long-term network event tracking on modest hardware with 20,000 inserts per second.
Contribution
The paper presents TWIAD, a novel database system leveraging write-optimized data structures for efficient long-term network event storage and analysis.
Findings
Achieved 20,000 inserts per second on a desktop system
Demonstrated feasibility of long-term network event tracking with low-cost hardware
Showed write-optimized data structures significantly improve data ingestion rates
Abstract
Access to network traffic records is an integral part of recognizing and addressing network security breaches. Even with the increasing sophistication of network attacks, basic network events such as connections between two IP addresses play an important role in any network defense. Given the duration of current attacks, long-term data archival is critical but typically very little of the data is ever accessed. Previous work has provided tools and identified the need to trace connections. However, traditional databases raise performance concerns as they are optimized for querying rather than ingestion. The study of write-optimized data structures (WODS) is a new and growing field that provides a novel approach to traditional storage structures (e.g., B-trees). WODS trade minor degradations in query performance for significant gains in the ability to quickly insert more data elements,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Caching and Content Delivery · Internet Traffic Analysis and Secure E-voting
