Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

TL;DR

This paper introduces a real-time, streaming log-based botnet detection method using an optimized Lanczos algorithm to improve PCA efficiency, enabling more sensitive detection in sliding windows.

Contribution

It adapts the Lanczos method for online PCA, reducing computational complexity and enabling real-time botnet detection from streaming logs.

Findings

Lanczos method reduces detection time to 20-25% of traditional PCA.

The approach effectively detects botnets in sliding time windows.

Experiments demonstrate improved speed and sensitivity in botnet detection.

Abstract

Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.