gSPICE: Model-Based Event Shedding in Complex Event Processing

TL;DR

This paper introduces gSPICE, a novel model-based black-box load shedding method for complex event processing systems that predicts event importance using probabilistic models and machine learning, improving result quality under overload conditions.

Contribution

It proposes a new set of features and a probabilistic model for event utility prediction, outperforming existing black-box load shedding approaches in maintaining QoR.

Findings

Outperforms state-of-the-art load shedding methods in most cases

Uses Zobrist hashing and machine learning for utility prediction

Validated on synthetic and real-world datasets

Abstract

Overload situations, in the presence of resource limitations, in complex event processing (CEP) systems are typically handled using load shedding to maintain a given latency bound. However, load shedding might negatively impact the quality of results (QoR). To minimize the shedding impact on QoR, CEP researchers propose shedding approaches that drop events/internal state with the lowest importances/utilities. In both black-box and white-box shedding approaches, different features are used to predict these utilities. In this work, we propose a novel black-box shedding approach that uses a new set of features to drop events from the input event stream to maintain a given latency bound. Our approach uses a probabilistic model to predict these event utilities. Moreover, our approach uses Zobrist hashing and well-known machine learning models, e.g., decision trees and random forests, to handle the predicted event utilities. Through extensive evaluations on several synthetic and two real-world datasets and a representative set of CEP queries, we show that, in the majority of cases, our load shedding approach outperforms state-of-the-art black-box load shedding approaches, w.r.t. QoR.