Intrusion Detection at Scale with the Assistance of a Command-line Language Model

TL;DR

This paper presents a large-scale AI-based intrusion detection system utilizing a pre-trained language model trained on tens of millions of command lines, demonstrating effectiveness on extensive datasets for enterprise security.

Contribution

It introduces a novel large-scale language model for intrusion detection trained on massive command line data, addressing scalability and generalization issues of prior methods.

Findings

Effective detection on 30 million training samples

High accuracy on 10 million test samples

Scalable solution for enterprise security environments

Abstract

Intrusion detection is a long standing and crucial problem in security. A system capable of detecting intrusions automatically is on great demand in enterprise security solutions. Existing solutions rely heavily on hand-crafted rules designed by security operators, which suffer from high false negative rates and poor generalization ability to new, zero-day attacks at scale. AI and machine learning offer promising solutions to address the issues, by inspecting abnormal user behaviors intelligently and automatically from data. However, existing learning-based intrusion detection systems in the literature are mostly designed for small data, and they lack the ability to leverage the power of big data in cloud environments. In this paper, we target at this problem and introduce an intrusion detection system which incorporates large-scale pre-training, so as to train a large language model based on tens of millions of command lines for AI-based intrusion detection. Experiments performed on 30 million training samples and 10 million test samples verify the effectiveness of our solution.