Forming IDEAS Interactive Data Exploration & Analysis System

TL;DR

This paper introduces IDEAS, an interactive system designed to enhance cyber security analysts' ability to perform deep data analysis, thereby improving detection of complex attacks through scientific inquiry and better decision support.

Contribution

The paper presents the design, implementation, and application of IDEAS, a modular, scalable system that enables more effective data exploration and analysis for cyber security operations.

Findings

Demonstrated three real-world use cases driving system design.

Proposed a scalable software architecture for data analysis.

Outlined plans for deployment in security operations.

Abstract

Modern cyber security operations collect an enormous amount of logging and alerting data. While analysts have the ability to query and compute simple statistics and plots from their data, current analytical tools are too simple to admit deep understanding. To detect advanced and novel attacks, analysts turn to manual investigations. While commonplace, current investigations are time-consuming, intuition-based, and proving insufficient. Our hypothesis is that arming the analyst with easy-to-use data science tools will increase their work efficiency, provide them with the ability to resolve hypotheses with scientific inquiry of their data, and support their decisions with evidence over intuition. To this end, we present our work to build IDEAS (Interactive Data Exploration and Analysis System). We present three real-world use-cases that drive the system design from the algorithmic capabilities to the user interface. Finally, a modular and scalable software architecture is discussed along with plans for our pilot deployment with a security operation command.