Detecting Novel Processes with CANDIES -- An Holistic Novelty Detection Technique based on Probabilistic Models

TL;DR

CANDIES is a holistic novelty detection method that uses probabilistic models to identify new processes in technical systems, especially in high- and low-density regions, with applications demonstrated in intrusion detection.

Contribution

The paper introduces CANDIES, a novel approach combining mixture models and specialized detection strategies for high- and low-density regions, advancing novelty detection in complex systems.

Findings

Effective detection of novel processes in high-density regions.

Successful application to intrusion detection benchmarks.

Improved detection accuracy over existing methods.

Abstract

In this article, we propose CANDIES (Combined Approach for Novelty Detection in Intelligent Embedded Systems), a new approach to novelty detection in technical systems. We assume that in a technical system several processes interact. If we observe these processes with sensors, we are able to model the observations (samples) with a probabilistic model, where, in an ideal case, the components of the parametric mixture density model we use, correspond to the processes in the real world. Eventually, at run-time, novel processes emerge in the technical systems such as in the case of an unpredictable failure. As a consequence, new kinds of samples are observed that require an adaptation of the model. CANDIES relies on mixtures of Gaussians which can be used for classification purposes, too. New processes may emerge in regions of the models' input spaces where few samples were observed before (low-density regions) or in regions where already many samples were available (high-density regions). The latter case is more difficult, but most existing solutions focus on the former. Novelty detection in low- and high-density regions requires different detection strategies. With CANDIES, we introduce a new technique to detect novel processes in high-density regions by means of a fast online goodness-of-fit test. For detection in low-density regions we combine this approach with a 2SND (Two-Stage-Novelty-Detector) which we presented in preliminary work. The properties of CANDIES are evaluated using artificial data and benchmark data from the field of intrusion detection in computer networks, where the task is to detect new kinds of attacks.