Feature Extraction and Feature Selection: Reducing Data Complexity with Apache Spark

TL;DR

This paper presents a scalable approach for feature extraction and selection in security analytics of heterogeneous network data using Apache Spark's pyspark API, aiming to improve efficiency in cyber threat detection.

Contribution

The paper introduces a novel implementation of feature extraction and selection tailored for heterogeneous security data using Apache Spark, enhancing processing efficiency.

Findings

Efficient handling of heterogeneous data sources.

Implementation in Apache Spark with pyspark.

Improved processing speed for security analytics.

Abstract

Feature extraction and feature selection are the first tasks in pre-processing of input logs in order to detect cyber security threats and attacks while utilizing machine learning. When it comes to the analysis of heterogeneous data derived from different sources, these tasks are found to be time-consuming and difficult to be managed efficiently. In this paper, we present an approach for handling feature extraction and feature selection for security analytics of heterogeneous data derived from different network sensors. The approach is implemented in Apache Spark, using its python API, named pyspark.