EigenEvent: An Algorithm for Event Detection from Complex Data Streams in Syndromic Surveillance

TL;DR

EigenEvent is a novel algorithm for early detection of disease outbreaks in complex, multivariate, and seasonal data streams, improving false alarm rates over existing methods by analyzing data correlation structures.

Contribution

The paper introduces EigenEvent, a new eigenspace-based approach that detects both overall and dimension-level changes, outperforming WSARE in false alarm reduction.

Findings

EigenEvent achieves lower false alarm rates than WSARE.

The method effectively detects both overall and specific data changes.

Experimental results on benchmark data validate its superior performance.

Abstract

Syndromic surveillance systems continuously monitor multiple pre-diagnostic daily streams of indicators from different regions with the aim of early detection of disease outbreaks. The main objective of these systems is to detect outbreaks hours or days before the clinical and laboratory confirmation. The type of data that is being generated via these systems is usually multivariate and seasonal with spatial and temporal dimensions. The algorithm What's Strange About Recent Events (WSARE) is the state-of-the-art method for such problems. It exhaustively searches for contrast sets in the multivariate data and signals an alarm when find statistically significant rules. This bottom-up approach presents a much lower detection delay comparing the existing top-down approaches. However, WSARE is very sensitive to the small-scale changes and subsequently comes with a relatively high rate of false alarms. We propose a new approach called EigenEvent that is neither fully top-down nor bottom-up. In this method, we instead of top-down or bottom-up search, track changes in data correlation structure via eigenspace techniques. This new methodology enables us to detect both overall changes (via eigenvalue) and dimension-level changes (via eigenvectors). Experimental results on hundred sets of benchmark data reveals that EigenEvent presents a better overall performance comparing state-of-the-art, in particular in terms of the false alarm rate.