# Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

## TL;DR

This paper introduces a real-time, streaming log-based botnet detection method using an optimized Lanczos algorithm to improve PCA efficiency, enabling more sensitive detection in sliding windows.

## Contribution

It adapts the Lanczos method for online PCA, reducing computational complexity and enabling real-time botnet detection from streaming logs.

## Key findings

- Lanczos method reduces detection time to 20-25% of traditional PCA.
- The approach effectively detects botnets in sliding time windows.
- Experiments demonstrate improved speed and sensitivity in botnet detection.

## Abstract

Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.

---
Source: https://tomesphere.com/paper/1812.07810