Learning detectors of malicious web requests for intrusion detection in network traffic

TL;DR

This paper introduces a low-resource, behavior-based detection system for malicious web requests that effectively identifies threats like C&C, phishing, and click fraud with high precision, using only limited proxy log data.

Contribution

The paper presents a novel, generic classification system that detects malicious web requests using statistical features from proxy logs, enabling deployment on diverse security devices.

Findings

Achieved over 95% precision in detecting malicious flows and URLs.

Detected a significant number of new threats beyond signature-based methods.

Maintained low computational requirements suitable for wide deployment.

Abstract

This paper proposes a generic classification system designed to detect security threats based on the behavior of malware samples. The system relies on statistical features computed from proxy log fields to train detectors using a database of malware samples. The behavior detectors serve as basic reusable building blocks of the multi-level detection architecture. The detectors identify malicious communication exploiting encrypted URL strings and domains generated by a Domain Generation Algorithm (DGA) which are frequently used in Command and Control (C&C), phishing, and click fraud. Surprisingly, very precise detectors can be built given only a limited amount of information extracted from a single proxy log. This way, the computational requirements of the detectors are kept low which allows for deployment on a wide range of security devices and without depending on traffic context such as DNS logs, Whois records, webpage content, etc. Results on several weeks of live traffic from 100+ companies having 350k+ hosts show correct detection with a precision exceeding 95% of malicious flows, 95% of malicious URLs and 90% of infected hosts. In addition, a comparison with a signature and rule-based solution shows that our system is able to detect significant amount of new threats.