How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios

TL;DR

This paper systematically evaluates provenance-based intrusion detection systems in industrial scenarios, revealing challenges like poor portability and high false positives, and proposes solutions to improve their effectiveness and reduce manual effort.

Contribution

It is the first comprehensive evaluation of PIDSes in industrial contexts, identifying key challenges and proposing methods to mitigate false positives and enhance detection performance.

Findings

PIDSes face portability issues across hosts and platforms.

Detection performance against real-world attacks is low.

High false positive rates increase with benign activity complexity.

Abstract

Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.