DEHYDRATOR: Enhancing Provenance Graph Storage via Hierarchical Encoding and Sequence Generation

TL;DR

Dehydrator is a novel system that significantly reduces storage requirements for provenance graphs by employing hierarchical encoding and neural network-based query support, outperforming traditional databases.

Contribution

It introduces a hierarchical encoding approach and deep neural network support for efficient provenance graph storage and querying, addressing high storage overhead issues.

Findings

Reduces storage space by 84.55%.

Outperforms PostgreSQL, Neo4j, and Leonard in efficiency.

Evaluated on over one billion log entries.

Abstract

As the scope and impact of cyber threats have expanded, analysts utilize audit logs to hunt threats and investigate attacks. The provenance graphs constructed from kernel logs are increasingly considered as an ideal data source due to their powerful semantic expression and attack historic correlation ability. However, storing provenance graphs with traditional databases faces the challenge of high storage overhead, given the high frequency of kernel events and the persistence of attacks. To address this, we propose Dehydrator, an efficient provenance graph storage system. For the logs generated by auditing frameworks, Dehydrator uses field mapping encoding to filter field-level redundancy, hierarchical encoding to filter structure-level redundancy, and finally learns a deep neural network to support batch querying. We have conducted evaluations on seven datasets totaling over one billion log entries. Experimental results show that Dehydrator reduces the storage space by 84.55%. Dehydrator is 7.36 times more efficient than PostgreSQL, 7.16 times than Neo4j, and 16.17 times than Leonard (the work most closely related to Dehydrator, published at Usenix Security'23).