A recommender system for efficient discovery of new anomalies in large-scale access logs

TL;DR

Helios is a novel recommender system that efficiently discovers unknown anomalies in large-scale access logs with minimal supervision, significantly accelerating security policy management and anomaly detection.

Contribution

The paper introduces Helios, a new recommender system that detects unseen anomalies in access logs without prior user or item information, using a bootstrapping approach and rank statistics.

Findings

Helios analyzes 4.6 billion records in under 60 minutes.

It accelerates anomaly discovery by 1 to 3 orders of magnitude.

The system is flexible with customizable metrics and measurement fields.

Abstract

We present a novel, non-standard recommender system for large-scale security policy management(SPM). Our system Helios discovers and recommends unknown and unseen anomalies in large-scale access logs with minimal supervision and no starting information on users and items. Typical recommender systems assume availability of user- and item-related information, but such information is not usually available in access logs. To resolve this problem, we first use discrete categorical labels to construct categorical combinations from access logs in a bootstrapping manner. Then, we utilize rank statistics of entity rank and order categorical combinations for recommendation. From a double-sided cold start, with minimal supervision, Helios learns to recommend most salient anomalies at large-scale, and provides visualizations to security experts to explain rationale behind the recommendations. Our experiments show Helios to be suitable for large-scale applications: from cold starts, in less than 60 minutes, Helios can analyze roughly 4.6 billion records in logs of 400GB with about 300 million potential categorical combinations, then generate ranked categorical combinations as recommended discoveries. We also show that, even with limited computing resources, Helios accelerates unknown and unseen anomaly discovery process for SPM by 1 to 3 orders of magnitude, depending on use cases. In addition, Helios' design is flexible with metrics and measurement fields used for discoveries and recommendations. Overall, our system leads to more efficient and customizable SPM processes with faster discoveries of unseen and unknown anomalies.