Beyond Collection: Measuring the Detection Efficacy of Modern Security Logging Standards

TL;DR

This paper systematically evaluates the effectiveness of modern security logging standards using a novel automated framework, revealing significant differences and gaps in their ability to detect cyber exploits.

Contribution

It introduces a new scalable methodology for evaluating logging standards and provides evidence-based insights for security practitioners.

Findings

Significant differences in logging standards' ability to detect exploits.

Identification of critical gaps in current security logging frameworks.

Development of a reproducible evaluation methodology for exploit telemetry.

Abstract

Effective security logging is crucial for the timely and accurate detection of cyber threats; however, the relative effectiveness of various industry-standard logging frameworks remains understudied. This paper addresses this critical gap by presenting the first systematic evaluation of modern security logging standards utilizing a novel methodology built upon the automated Security Exploit Telemetry Collection (SETC) framework. SETC systematically generates reproducible exploit scenarios in containerized environments, collecting rich telemetry across multiple logging standards, including CIM (Common Information Model), OCSF (Open Cybersecurity Schema Framework), and ECS (Elastic Common Schema). The detection efficacy of each logging standard is quantified by measuring telemetry completeness and exploit detectability across standardized logs through detailed experiments involving 50 diverse remote code execution vulnerabilities. The resulting findings identify critical gaps and reveal significant differences in logging standards' abilities to capture key attack indicators. Our contributions include a novel evaluation methodology that enables scalable and reproducible analysis of exploit telemetry, as well as new findings that provide clear, evidence-based guidance for security practitioners to make informed decisions about adopting logging standards.