Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data

TL;DR

This paper presents a novel belief propagation framework for early detection of enterprise infections using large-scale log data, effectively identifying stealthy malware and APT attacks that evade traditional security tools.

Contribution

It introduces a new graph-theoretic belief propagation method for early infection detection, capable of operating with or without seed information, and demonstrates its effectiveness on real-world and simulated data.

Findings

High detection accuracy with low false positives and negatives.

Identified hundreds of malicious domains missed by existing security tools.

Effective in detecting stealthy infections with minimal prior information.

Abstract

Recent years have seen the rise of more sophisticated attacks including advanced persistent threats (APTs) which pose severe risks to organizations and governments by targeting confidential proprietary information. Additionally, new malware strains are appearing at a higher rate than ever before. Since many of these malware are designed to evade existing security products, traditional defenses deployed by most enterprises today, e.g., anti-virus, firewalls, intrusion detection systems, often fail at detecting infections at an early stage. We address the problem of detecting early-stage infection in an enterprise setting by proposing a new framework based on belief propagation inspired from graph theory. Belief propagation can be used either with "seeds" of compromised hosts or malicious domains (provided by the enterprise security operation center -- SOC) or without any seeds. In the latter case we develop a detector of C&C communication particularly tailored to enterprises which can detect a stealthy compromise of only a single host communicating with the C&C server. We demonstrate that our techniques perform well on detecting enterprise infections. We achieve high accuracy with low false detection and false negative rates on two months of anonymized DNS logs released by Los Alamos National Lab (LANL), which include APT infection attacks simulated by LANL domain experts. We also apply our algorithms to 38TB of real-world web proxy logs collected at the border of a large enterprise. Through careful manual investigation in collaboration with the enterprise SOC, we show that our techniques identified hundreds of malicious domains overlooked by state-of-the-art security products.