Towards an Open Format for Scalable System Telemetry

TL;DR

SysFlow is a new open data format for system telemetry that offers comprehensive visibility into system activities, reduces storage needs, and enhances security analytics for big data environments.

Contribution

Introduces SysFlow, a compact, open, flow-centric data format for system telemetry that improves visibility, reduces storage, and supports advanced security analytics.

Findings

SysFlow traces are orders of magnitude smaller than existing approaches.

Enables deeper attack chain analysis and forensic investigations.

Supports long-term data archival for cyber threat detection.

Abstract

A data representation for system behavior telemetry for scalable big data security analytics is presented, affording telemetry consumers comprehensive visibility into workloads at reduced storage and processing overheads. The new abstraction, SysFlow, is a compact open data format that lifts the representation of system activities into a flow-centric, object-relational mapping that records how applications interact with their environment, relating processes to file accesses, network activities, and runtime information. The telemetry format supports single-event and volumetric flow representations of process control flows, file interactions, and network communications. Evaluation on enterprise-grade benchmarks shows that SysFlow facilitates deeper introspection into attack kill chains while yielding traces orders of magnitude smaller than current state-of-the-art system telemetry approaches -- drastically reducing storage requirements and enabling feature-filled system analytics, process-level provenance tracking, and long-term data archival for cyber threat discovery and forensic analysis on historical data.