NetMemex: Providing Full-Fidelity Traffic Archival

TL;DR

NetMemex is a high-performance, cost-effective system for full-fidelity network traffic archival that enables detailed forensic analysis and research by efficiently storing and retrieving complete packet data at near-Gbps speeds.

Contribution

It introduces a novel system combining data compression, optimized data layout, and high-speed storage to achieve full traffic archival with fast query capabilities on commodity hardware.

Findings

Records full-fidelity traffic at near-Gbps rates

Handles up to 90.1K queries/second

Maintains low storage costs comparable to traditional solutions

Abstract

NetMemex explores efficient network traffic archival without any loss of information. Unlike NetFlow-like aggregation, NetMemex allows retrieving the entire packet data including full payload, which makes it useful in forensic analysis, networked and distributed system research, and network administration. Different from packet trace dumps, NetMemex performs sophisticated data compression for small storage space use and optimizes the data layout for fast query processing. NetMemex takes advantage of high-speed random access of flash drives and inexpensive storage space of hard disk drives. These efforts lead to a cost-effective yet high-performance full traffic archival system. We demonstrate that NetMemex can record full-fidelity traffic at near-Gbps rates using a single commodity machine, handling common queries at up to 90.1 K queries/second, at a low storage cost comparable to conventional hard disk-only traffic archival solutions.