Improving Outbreak Detection with Stacking of Statistical Surveillance Methods

TL;DR

This paper explores enhancing outbreak detection by stacking statistical surveillance methods using machine learning, focusing on p-values and additional features to improve detection accuracy while managing false alarms.

Contribution

It introduces a novel fusion approach utilizing p-values and extra features, along with a new performance measure, to improve outbreak detection accuracy.

Findings

Fusion based on binary outputs can worsen performance.

Using p-values and additional features shows promise.

New measure better captures false alarm trade-offs.

Abstract

Epidemiologists use a variety of statistical algorithms for the early detection of outbreaks. The practical usefulness of such methods highly depends on the trade-off between the detection rate of outbreaks and the chances of raising a false alarm. Recent research has shown that the use of machine learning for the fusion of multiple statistical algorithms improves outbreak detection. Instead of relying only on the binary output (alarm or no alarm) of the statistical algorithms, we propose to make use of their p-values for training a fusion classifier. In addition, we also show that adding additional features and adapting the labeling of an epidemic period may further improve performance. For comparison and evaluation, a new measure is introduced which captures the performance of an outbreak detection method with respect to a low rate of false alarms more precisely than previous works. Our results on synthetic data show that it is challenging to improve the performance with a trainable fusion method based on machine learning. In particular, the use of a fusion classifier that is only based on binary outputs of the statistical surveillance methods can make the overall performance worse than directly using the underlying algorithms. However, the use of p-values and additional information for the learning is promising, enabling to identify more valuable patterns to detect outbreaks.

Figures

Tables

Table 1. Table 1 . Comparison of including or not including the mean in the data for ML algorithms: overall denotes all 42 42 42 test cases, { ( ¬ ) ⁡ T , ( ¬ ) ⁡ S ​ 1 , ( ¬ ) ⁡ S ​ 2 } 𝑇 𝑆 1 𝑆 2 \{(\neg)T,(\neg)S1,(\neg)S2\} only cases (not) containing trend, annual/biannual seasonality, respectively. Each particular subset, fulfilling constraints on seasonality and trend, include 6 6 6 test cases.

Approach	Overall	$\{\neg T,\neg S1,\neg S2\}$	$\{\neg T,S1,\neg S2\}$	$\{\neg T,S1,S2\}$	$\{T,\neg S1,\neg S2\}$	$\{T,S1,\neg S2\}$	$\{T,S1,S2\}$
S($\neg \mu$, $O_0$, $0$)	3.429	3.714	3.571	3.000	3.429	3.286	3.571
S($\mu$, $O_0$, $0$)	3.357	2.571	3.286	3.571	3.571	3.714	3.429
P($\neg \mu$, $O_0$, $0$)	1.905	2.571	1.857	2.000	1.714	1.714	1.571
P($\mu$, $O_0$, $0$)	1.310	1.143	1.286	1.429	1.286	1.286	1.429

Table 2. Table 2 . Comparison of different window sizes for the data (including the mean and using the labeling O 0 subscript 𝑂 0 O_{0} ).

Approach	Overall	$\{\neg T,\neg S1,\neg S2\}$	$\{\neg T,S1,\neg S2\}$	$\{\neg T,S1,S2\}$	$\{T,\neg S1,\neg S2\}$	$\{T,S1,\neg S2\}$	$\{T,S1,S2\}$
S($\neg \mu$,$O_0$,0)	9.738	9.000	9.571	8.286	11.143	10.571	9.857
S($\neg \mu$,$O_0$,1)	8.738	8.857	7.000	9.000	7.571	8.857	11.143
S($\neg \mu$,$O_0$,2)	10.762	10.571	10.857	10.714	10.571	10.143	11.714
S($\neg \mu$,$O_0$,4)	11.310	11.429	11.714	11.714	10.857	12.000	10.143
S($\neg \mu$,$O_0$,6)	11.619	12.714	12.286	10.286	11.571	11.857	11.000
S($\neg \mu$,$O_0$,8)	11.548	11.143	11.571	12.000	10.857	12.000	11.714
S($\neg \mu$,$O_0$,12)	11.929	12.143	12.143	13.000	11.714	11.571	11.000
P($\neg \mu$,$O_0$,0)	5.000	5.714	5.000	5.714	4.429	3.714	5.429
P($\neg \mu$,$O_0$,1)	3.405	3.143	2.571	4.571	3.286	4.286	2.571
P($\neg \mu$,$O_0$,2)	4.381	5.000	4.714	4.000	4.571	4.571	3.429
P($\neg \mu$,$O_0$,4)	4.667	4.143	5.000	4.000	5.143	5.286	4.429
P($\neg \mu$,$O_0$,6)	4.310	5.000	4.429	3.857	3.857	3.857	4.857
P($\neg \mu$,$O_0$,8)	4.000	3.000	4.000	4.714	5.000	3.857	3.429
P($\neg \mu$,$O_0$,12)	3.595	3.143	4.143	3.143	4.429	2.429	4.286

Table 3. Table 3 . Comparison of the different labeling strategies for the epidemics (not using the average and w = 0 𝑤 0 w=0 ).

Approach	Overall	$\{\neg T,\neg S1,\neg S2\}$	$\{\neg T,S1,\neg S2\}$	$\{\neg T,S1,S2\}$	$\{T,\neg S1,\neg S2\}$	$\{T,S1,\neg S2\}$	$\{T,S1,S2\}$
S($\neg \mu$, $O_0$, $0$)	6.476	6.286	5.571	4.857	7.143	7.571	7.429
S($\neg \mu$, $O_1$, $0$)	6.738	7.286	6.714	6.286	6.429	7.000	6.714
S($\neg \mu$, $O_2$, $0$)	5.738	6.286	5.714	5.286	5.429	6.000	5.714
S($\neg \mu$, $O_3$, $0$)	5.524	5.286	5.143	5.429	6.000	5.429	5.857
P($\neg \mu$, $O_0$, $0$)	3.762	3.857	3.857	2.714	4.857	4.000	3.286
P($\neg \mu$, $O_1$, $0$)	3.262	2.857	4.143	4.857	2.429	2.429	2.857
P($\neg \mu$, $O_2$, $0$)	2.690	3.143	3.000	3.286	2.714	2.143	1.857
P($\neg \mu$, $O_3$, $0$)	1.810	1.000	1.857	3.286	1.000	1.429	2.286

Table 4. Table 4 . Comparison of the standard fusion, the p 𝑝 p -value fusion and each individual statistical algorithm.

Approach	Overall	$\{\neg T,\neg S1,\neg S2\}$	$\{\neg T,S1,\neg S2\}$	$\{\neg T,S1,S2\}$	$\{T,\neg S1,\neg S2\}$	$\{T,S1,\neg S2\}$	$\{T,S1,S2\}$
C1	5.381	6.429	5.429	4.143	5.714	5.714	4.857
C2	4.810	4.571	4.000	4.286	5.857	5.286	4.857
C3	4.690	5.429	4.571	4.286	4.857	4.429	4.571
Bayes	2.595	4.000	3.143	2.571	1.571	1.714	2.571
RKI	3.619	3.571	2.857	4.571	3.714	3.286	3.714
S($\mu$, $O_3$, $1$)	5.238	3.000	6.000	5.714	4.714	5.857	6.143
P($\mu$, $O_3$, $1$)	1.667	1.000	2.000	2.429	1.571	1.714	1.286

Equations

$$\mu (t)=\frac{1}{m}\sum _{i=1}^m{c}_{t-i}$$

$$c_t\stackrel{\text{C1}}{\sim }N(\mu (t),{\sigma }^2(t))c_t\stackrel{\text{C2}}{\sim }N(\mu (t-2),{\sigma }^2(t-2))$$

$$\left[\frac{c_t-\mu (t-2)}{\sqrt{{\sigma }^2(t-2)}}-\sum _{i=1}^2\max (0,\frac{c_{t-i}-\mu (t-2-i)}{\sqrt{{\sigma }^2(t-2-i)}}-1)\right]\stackrel{\text{C3}}{\sim }N(0,1)$$

$$c_t\stackrel{\text{Bayes}}{\sim }NB(m\cdot \mu (t)+\frac{1}{2},\frac{m}{m+1})$$

$$c_t\stackrel{\text{RKI}}{\sim }\{\begin{array}{ll}Poisson(\lfloor \mu (t)\rfloor +1),& \text{if }\mu (t)\le 20\\ N(\mu (t),{\sigma }^2(t)),& \text{otherwise}\end{array}$$

$${\text{pAUC}}_e=\frac{{\int }_0^{e}ROC(f)df}{e}$$

Full text

††copyright: acmcopyright††journalyear: 2019††conference: epiDAMINK 2019: Epidemiology meets Data Mining and Knowledge discovery, Workshop held in conjuction with ACM SIGKDD 2019; August 05, 2019; Anchorage, Alaska - USA

1. Introduction

The early detection of infectious disease outbreaks is of great significance for public health. The spread of such outbreaks could be diminished tremendously by applying control measures as early as possible, which indeed can save lives and reduce suffering (Noufaily et al., 2013). For that purpose, statistical algorithms have been developed to automate and improve outbreak detection. Such methods raise alarms in the case that an unusually high number of infections is detected which results in a further investigation by an epidemiologist (Jackson et al., 2007). Ideally, such algorithms are completely automated while still being able to be applied on a wide spectrum of different infections and syndromes (Noufaily et al., 2019). However, if not chosen wisely or configured properly, they may also raise many false alarms which can overwhelm the epidemiologist. In particular for large surveillance systems, where many time series for different diseases and different locations are monitored simultaneously, the false alarm rate is a major concern and therefore highly determines the practical usefulness of an outbreak detection method (Shmueli and Burkom, 2010). However, regulating the false alarm rate usually has an impact on the ability to detect outbreaks. To find a good trade-off between those measures is one of the major challenges in outbreak detection (Noufaily et al., 2013; Bédubourg and Le Strat, 2017).

Traditional outbreak detection methods rely on historic data to fit a parametric distribution which is then used to check the statistical significance of the current observation. Choosing the significance level for the statistical method beforehand makes the evaluation difficult. In line with Kleinman and Abrams (2006), we propose a method which uses the $p$-values of the statistical methods in order to evaluate their performance.

In particular, we propose a variant of Receiver Operating Characteristic (ROC) curves, which shows the false alarm rate on the $x$-axis and the detection rate—in contrast to the true positive rate—on the $y$-axis. By using the area under the partial ROC curve (Ma et al., 2013), we are able to obtain a measure for the performance of an algorithm satisfying a particular constraint on the false alarm rate (e.g. less than 1% false alarms). This criterion serves as the main measure for our evaluations and enables to analyze the trade-off between the false alarm rate and the detection rate of outbreak detection methods precisely.

Prior work on outbreak detection mainly focuses on forecasting the number of infections for a disease (e.g. (Farrow et al., 2017; Chakraborty et al., 2014)). However, only little research has been devoted to use supervised machine learning (ML) techniques for improving algorithms, which can raise alarms. Jafarpour et al. (2015) used Baysian networks to identify the determinants for detection performance to find appropriate algorithm configurations for outbreak detection methods. Furthermore, classification algorithms and voting schemes have been used for the fusion of outbreak detection methods on univariate time series (Texier et al., 2019; Jafarpour et al., 2013) as well as on multi-stream time series (Lau et al., 2008; Mnatsakanyan et al., 2009; Burkom et al., 2011). However, the examined approaches only rely on the binary output (alarm or no alarm) of the underlying statistical methods for the fusion which limits the information about a particular observation. Prior research in the area of ML has shown that more precise information of the underlying models improves the overall performance of the fusion (Ting and Witten, 1999). Therefore, we propose an approach for the fusion of outbreak detection methods which uses the $p$-values of the underlying statistical methods. Moreover, one can also incorporate different information for the outbreak detection (e.g., weather data, holidays, statistics about the data, …) by just augmenting the data with additional attributes. As a first step, we put our focus on improving the performance of outbreak detection methods using an univariate time series as the only source of information. Furthermore, the way outbreaks are labeled in the data also has a major influence on the learnability of outbreak detectors. Thus, we propose adaptions for the labeling of outbreaks in order to maximize the detection rate of ML algorithms.

2. Statistical Algorithms for Syndromic Surveillance

The key idea of our approach is to learn to combine predictions of commonly used statistical outbreak detection methods with a trainable ML algorithm. Thus, we first need to generate a series of aligned prediction vectors, each consisting of one entry for each method. This sequence can then be used for training the ML model.

Let us denote with $C=(c_{0},c_{1},\ldots,c_{n})\in\mathbb{N}^{n}$ the time series of infection counts for a particular disease. Many methods rely on a sliding window approach which uses the previous $m$ counts as reference values for fitting a particular parametric distribution. Therefore, the mean $\mu(t)$ and the variance $\sigma^{2}(t)$ can be computed over these $m$ reference values as follows:

[TABLE]

On the fitted distributions, a statistical significance test is performed in order to identify suspicious spikes. For the purpose of outbreak detection, we rely on one tailed-tests for the statistical algorithms in order to only capture the observation of unusual high number of infections. For a particular observed count $c_{t}$ and a fitted distribution $p(x)$, the $p$-value is computed as the probability $\int_{c_{t}}^{\infty}p(x)dx$ of observing $c_{t}$ or higher counts. Hence, small $p$-values represent uncommonly high counts of $c_{t}$. The sensitivity of raising an alarm is regulated by the significance level $\alpha$ and if the $p$-value is inferior to the threshold $\alpha$ an alarm is raised.

We have chosen to base our work on the following methods which are all implemented in the R package surveillance (Salmon et al., 2016):

**EARS C1: **

and EARS C2 are variants of the Early Aberration Reporting System (Hutwagner et al., 2003; Fricker Jr. et al., 2008) which rely on the assumption of a Gaussian distribution. The difference between C2 and C1 lies in the added gap of two time points between the reference values and the current observed count $c_{t}$, so that the distribution of $c_{t}$ are assumed as in the following:

[TABLE]

**EARS C3: **

combines the result of the C2 method over a period of three previous observations. For convenience of notation, the incidence counts $c_{t}$ for the C3 method are transformed according to the statistics so that it fits to the normal distribution.

[TABLE]

Despite the inaccurate assumption of the Gaussian distribution for low counts, the EARS variants are often included in comparative studies due to its simplicity and still serves as competitive baseline (Bédubourg and Le Strat, 2017; Fricker Jr. et al., 2008; Hutwagner et al., 2005).

**Bayes method.: **

In contrast to the family of C-algorithms, the Bayes algorithm relies on the assumption of a negative binomial distribution:

[TABLE]

**RKI method.: **

Since the Gaussian distribution is not suitable for count data with a low mean, the RKI algorithm, as implemented by Salmon et al. (2016), assumes a Poisson distribution:

[TABLE]

They all have in common that they require comparably little historic data on their own, which allows us to train the ML method on longer sequences. Moreover, such methods are universally applicable and serve as drop-in approaches for surveillance systems since they only rely on the detection of a local increase in incidents without the need to capture effects like seasonality and trend.

3. Fusion Methods

The combination of information from several sources in order to obtain a unified picture is known as fusion (Khaleghi et al., 2013). Classifier fusion is a special case which combines the outputs of multiple classifiers in order to improve classification performance. In our context, the statistical algorithms for syndromic surveillance can be seen as classifiers, each classifying the current observation into the classes alarm or no alarm. A straight-forward way for combining the predictions of multiple outbreak detection methods is to simply vote and follow the majority prediction. A more sophisticated approach consists of training a classifier that uses the predictions of the detection methods as input, and is trained on the desired output, a technique that is known in ML as stacking (Wolpert, 1992).

Recent work in the area of outbreak detection and fusion has focused on fusing the information obtained by simultaneously monitoring multiple time series for a particular disease. Lau et al. (2008) have shown that the performance of statistical algorithms can already be improved by combining them with simple voting schemes. Mnatsakanyan et al. (2009) could further improve the performance using Bayesian networks and including further information about the patients (e.g., age) as additional attributes. Moreover, Burkom et al. (2011) have used a hierarchy of Bayesian networks in order to incorporate additional information about health surveillance data and environmental sensors. However, all of these fusion methods aim to capture the degree of dependence between the monitored time series relying on spatial correlations.

Only little research has been devoted to improving the performance of statistical algorithms on univariate time series. In particular, Texier et al. (2019) have used the ML technique hierarchical mixture of experts (Jordan and Jacobs, 1994) to combine the output of the methods from EARS. However, the authors note that all algorithms rely on the assumption of a Gaussian distribution, which limits their diversity. In contrast, Jafarpour et al. (2013) have used a variety of classification algorithms (logistic regression, CART and Baysian Networks) for the fusion of outbreak detection methods. As underlying statistical algorithms they have used the Cumulative Sum (CUSUM), two Exponential Weighted Moving Average algorithms, the EARS methods (C1,C2,C3) and the Farrington algorithm (Noufaily et al., 2013). In general, the results of Texier et al. (2019) and Jafarpour et al. (2013) indicate that ML improves the ability to detect outbreaks while simple voting schemes (e.g. weighted voting and majority vote) did not perform well. Moreover, the algorithms have not been evaluated with respect to data which include seasonality and trend.

4. Fusion with Augmented Stacking

In this work, we show that the availability of additional information can further improve the performance of the fusion classifier. Therefore, we first propose to use $p$-values of the statistical methods for the fusion in order to include information about the certainty of an alarm, and then show how to add additional external information to the learning process of the ML algorithm. Finally, we investigate different variants for labeling outbreaks.

4.1. Fusion with $p$-values

Given base estimators $g_{1}(x),\ldots,g_{K}(x)$, a fusion combiner is a function $h(g_{1}(x),\ldots,g_{K}(x))$ that combines the predictions of the base functions. In the simple case of binary voting, i.e., $g_{i}(x)\in\{0,1\}$, the combiner $h(x)=\frac{1}{K}\sum_{i}g_{i}(x)$ with a threshold of $0.5$ would model the majority rule. In stacking the function $h:X^{K}\xrightarrow{}O$ is learned by training a machine learning classifier on a set of previous observations $(g_{1}(x_{1}),\ldots,g_{K}(x_{1}))$, …, $(g_{1}(x_{n}),\ldots,g_{K}(x_{n}))$ –derived from applying $g_{i}$ on $x_{t}$– with associated targets $o_{1},\ldots,o_{n}\in O$. We refer to this as the training set in contrast to the evaluation set, which contains new, unseen observations. In outbreak detection, the instances $x_{t}$ correspond to the points in the time series $C$ of infection counts $c_{t}$ and $o_{t}\in\{0,1\}$ denotes the labelling of a time point as belonging to an outbreak (1) or not (0).

Previous approaches (Texier et al., 2019; Jafarpour et al., 2013) used the binary alarms ({0,1}) of base outbreak detectors. In this work instead, we propose to base our stacking model on the $p$-values, i.e., $g_{i}(x)\in[0,1]$, provided by the underlying statistical approaches (cf. Sec. 2). In fact, the $p$-values can directly be seen as the certainty of currently observing an outbreak, enabling the learning algorithm to make use of the base estimations in a much more fine grained way. This information is otherwise lost when using binary alarms, which are indeed obtained by just applying a fixed threshold on the computed $p$-values. In addition to the circumvented difficulty of tuning such threshold, previous studies on stacking have shown empirically that using the raw predictions can improve over the discretized option (Ting and Witten, 1999).

Figure 1 visualizes an example on how the data for the learning algorithm is created by using the $p$-values of the statistical algorithms Bayes and RKI. The columns RKIt and Bayest represent the computed $p$-values for the current observation while the other columns (meant, RKIt-1 and Bayest-1) represent additional information explained in the following section.

4.2. Additional Features

The use of a trainable fusion method allows us to include additional information which can help to decide whether a given alarm should be raised or not. As additional features, we propose to include the mean of the counts over the last $m$ time points (the same number of time points as used by the statistical methods), which can give us evidence about the reliability of a particular outcome. For example, the assumption of a Gaussian distribution for a low mean of count data ($\leq 20$) is known to be imprecise. Therefore, a learning algorithm might induce in this scenario that the $p$-values of the statistical methods C1, C2 and C3 may not be trustworthy. Moreover, under the assumption that a time series is stationary an unusual high mean can also be a good indicator to detect an outbreak, especially in the case that an outbreak arises slowly over time. The column $\text{mean}_{t}$ in Figure 1 illustrates how the mean over the last four observed counts ($m=4$) is added as an additional feature.

Finally, we also include the output of the statistical methods for previous time points in a window of a user-defined size $w$ as additional features.

For the example in Figure 1, we have used a window size of one ($w=1$) which includes the previous output of both statistical algorithms.

4.3. Modelling the Output Labels for Learning

A major challenge for ML algorithms is that the duration of an outbreak period is not clearly defined (Shmueli and Burkom, 2010). A simple strategy—which we refer to as $O_{0}$—is to label all time points positive as long as cases for the particular epidemic are reported (e.g. time points prior to the peak of an outbreak and a few time points after the peak). In this case, the goal of the learning algorithm is to predict most time points in an ongoing epidemic as positive, regardless of their time stamp. Indeed, our early results indicate that the predictor learns to recognize the fading-out of an outbreak (e.g. weeks 40 to 42 in Figure 1). This is due to the fact that the peak of the outbreak is included in the reference values which results in a considerably high mean $\mu(t)$ for the significance test. Because of this, unusually high $p$-values are generated for the counts after the peak, which provide sufficient evidence for the stacking algorithm to raise an alarm. However, this also increases the number of false alarms as the ML approach learns to raise alarms when the count is decreasing outside an epidemic period.

To avoid this, we propose three adaptations of $O_{0}$: $O_{1}$ labels all time points until the peak (the point with maximum number of counts during the period) as positive. $O_{2}$ instead skips the time points whose count is decreasing compared to the immediate previous count (i.e., it labels all increasing counts until reaching the peak). Finally, $O_{3}$ labels only the peak of the outbreak as positive. Figure 1 visualizes an example outbreak with the corresponding different options to label the epidemic period on the top-left.

5. Evaluation Measures

Instead of manually adjusting the $\alpha$ parameter of the statistical methods and examining the results individually, which is mostly done in previous works, we propose to evaluate the $p$-value as it is done by Kleinman and Abrams (2006). In particular, the $p$-value can be interpreted as a score, which sorts examples according to their degree to which they indicate an alarm. This allows us to analyze an algorithm with ROC curves (Fawcett, 2006). A ROC curve can be used to examine the trade-off between the true positive rate (i.e., the probability of raising an alarm in case of an actual outbreak) and the false alarm rate (i.e., the probability of falsely raising an alarm when no outbreak is ongoing). In order to only focus on high specificity results (e.g., with a false alarm rate below $1\%$), which is of major importance for many medical applications, we only consider partial ROC curves. By using the partial area under the ROC curve as proposed in (Ma et al., 2013), we obtain a simple measure to evaluate the performance of an algorithm, satisfying particular constraint on the false alarm rate. We refer to this measure as pAUCe where the parameter $e$ defines the maximum allowed false alarm rate to be considered. It is computed as

[TABLE]

where $ROC(f)$ denotes the true positive rate given a false alarm rate of $f$.

However, alarms raised in cases when the epidemic has already been detected are typically not very decisive and informative anymore. To incorporate this, we consider the detection rate, which represents the proportion of recognized outbreaks (i.e., the outbreaks in which at least one alarm is raised during their activity). Following Kleinman and Abrams (2006) and Jafarpour et al. (2013), we therefore use a ROC curve-like representation with the detection rate on the $y$-axis instead of the true positive rate, and use dAUCe to refer to the partial area under this curve. Figure 2 shows an example of the ROC-curve like representation and visualizes the area of dAUC1%. Kleinman and Abrams (2006) proposed to use weighted ROC curves to also incorporate the influence of the measure timeliness (mean time to detect an outbreak). However, we argue that the weighing with the timeliness introduces a trade-off (importance of timeliness over detection rate) and a loss in interpretability of the absolute numbers.

6. Evaluation

The key aspect of our experimental evaluation is to demonstrate that the fusion of $p$-values leads to a further improvement in performance compared to only using the binary output of the statistical algorithms. However, for a deeper understanding of our proposed approaches, we first performed experiments to evaluate the influence of including additional features for the stacking in Section 6.2, followed by an analysis of adapting the labeling for the learning in Section 6.3. Finally, using the obtained knowledge about the effect of the proposed techniques, we compare them with the underlying statistical algorithms in Section 6.4, which represents our main result.

6.1. Experimental Setup

As an implementation baseline for the statistical methods, we have used the R package surveillance (Salmon et al., 2016) and adapted the implementation of the methods EARS (C1, C2 and C3), Bayes and RKI in order to also return $p$-values. All methods use the previous $7$ time points as reference values, which is the standard configuration. For the ML part, we rely on the Python library scikit-learn (Pedregosa et al., 2011). To keep the evaluation simple, we use a random forest classifier. Basically, it learns an ensemble of randomized decision trees, which has proven to be robust in performance theoretically and practically (Fernández-Delgado et al., 2014; Wyner et al., 2017). Each model is composed of 100 decision trees with a minimum number of instances per leaf of $5$ and default settings otherwise. To allow comparability between the fusion methods, we also evaluated the approach which only combines the binary outputs of the statistical methods as proposed in (Texier et al., 2019; Jafarpour et al., 2013) and which we refer to as the standard fusion. Our preliminary experiments have shown that $\alpha=0.5\%$ for the underlying statistical methods performs best for this fusion approach. For all evaluations, we focused on our proposed evaluation measure $dAUC_{1\%}$ where we fixed the constraint on the false alarm rate to be less than $1$%.

Our evaluation is based on synthetic data which have been proposed by Noufaily et al. (2013). In total $42$ different test cases are used which reflect a wide range of application scenarios allowing to analyze the effects of trend (T), seasonality (S1) and biannual seasonality (S2) explicitly. For each parameter configuration $100$ time series are generated, each containing a total of $624$ weeks. Following Noufaily et al. (2013), the last 49 weeks of each time series serve as evaluation data which include exactly one outbreak whereas the first $575$ weeks contain four outbreaks and represents the so called baseline data. Each outbreak starts at a randomly drawn week and the number of cases per outbreak is generated with a Poisson distribution with the mean equal to a constant $k$ times the standard deviation of the counts observed at the starting week. The outbreak cases are then distributed over time using a log-normal distribution with mean [math] and standard deviation $0.5$. We evaluated each stacking configuration separately for each test case using the baseline data of the 100 time series for training (in total $57.500$ weeks including $400$ outbreaks) and the remaining $4.900$ weeks for testing ($100$ outbreaks), respectively. The statistical methods were applied separately for each time series in order to obtain the $p$-values as inputs for the learner as well as the predictions on the evaluation set.

Instead of reporting the average over $dAUC_{1\%}$ scores, which could have different scales for different test cases, we determined a ranking over the compared methods for each test case. Afterwards, each method’s rank is averaged across the evaluated test cases to obtain an overall rank. In order to evaluate the effects of trend and seasonality explicitly, we average the rankings only over the test cases which include these effects. To differentiate between our proposed approaches, we use the notation $M(a,o,w)$ where $M\in\{P,S\}$ specifies whether $p$-value fusion (P) or the standard fusion (S) has been used, $a\in\{\neg\mu,\mu\}$ whether the mean is included, $o\in\{O_{0},O_{1},O_{2},O_{3}\}$ which labeling for the learning, and $w\in\{0,1,\ldots,12\}$ the window size which has been used for the evaluation. In total, we tested $192$ configurations from which we compare only a small subset , respectively, depending on the analyzed aspect.

6.2. Evaluation of Additional Features

The first aspect to review concerns the inclusion of the mean count over the last seven time points. Therefore, we have analyzed the effect of this feature independent of the other parameters using $O_{0}$ for the labeling of the outbreak and window size $w=0$. The results for the average rank are displayed in Table 1. Comparing the standard to the $p$-value fusion method reveals a beneficial effect especially for the $p$-value approach, for which the variant including the mean achieves an average rank of $1.31$ over $1.91$. In contrast, the average ranks of $3.36$ over $3.43$ for the standard method not only shows that there are issues regarding the usage of the mean for some of the test case configurations, but also the substantial gap between using the binary outputs and the more fine-grained $p$-values. A closer examination reveals that the best improvement for both fusion methods can be achieved on time series without trend and seasonality. By adding effects like trend and seasonality, the mean changes over time, making it difficult for the learning algorithm to use this information. In contrast to the standard fusion, the $p$-value fusion method still enhances by including the mean over the previous time points.

The observation that the $p$-value fusion method is superior to the standard fusion can also be seen when comparing different window sizes. The results of this experiment, using $O_{0}$ for the labeling of the outbreak and not including the mean, are displayed in Table 2. In particular, no window configuration of the standard fusion method can outperform any of the $p$-value configurations with respect to the average rank. Overall, a window size of $1$ performed best for both fusion approaches. Being able to compare to the most immediate previous output of the underlying statistical algorithms seems to make it easier to detect anomalies. In contrast, larger window sizes harm the overall performance, which suggests that the additional information is not relevant for detecting sudden changes and rather confuses the learner. Interestingly, on certain combinations of trend and seasonality a larger window size for the $p$-value fusion method seems to be beneficial. Actually, the increase of the window size also results in taking a further look back in the past allowing to detect effects like trend and seasonality achieving good results on the test cases which only contain biannual seasonality. However, the observed results for larger window sizes are inconsistent across the different test cases, making it difficult to draw valid conclusions.

6.3. Evaluation of the Labeling Adaptions

In addition to augmenting the input data, we have evaluated the effect of adapting the labeling of the epidemic period for the training of the stacking algorithm. The comparison shown in Table 3 was performed without the augmentation.

In general, we can observe that by narrowing the labeling of the outbreak on particular events (i.e., $O_{1}$, $O_{2}$ or $O_{3}$) a better performance can be achieved. This effect is clearly visible for the $p$-value fusion method and less obvious for the standard fusion method, for which the adaption $O_{1}$ seems to be an exception. In particular, learning only the peaks ($O_{3}$) achieved the best results for both fusion approaches. The benefit of this variant is that the learner can actually focus on the identification of strong and sudden peaks which is indeed the main goal of outbreak detection. However, in case of biannual seasonality the frequent change of the counts over the season results in many random peaks which apparently makes it difficult for the stacking approach to distinguish between an epidemic peak and a peak caused by random effects. On the test cases without trend ($\{\neg{T},S1,S2\}$) outbreaks are better identifiable by also including the fading of the outbreak ($O_{0}$), whereas on the test cases which contain trend ($\{T,S1,S2\}$) the best option seems to be $O_{2}$, which only includes only the increasing counts until the peak of the outbreak is reached ($O_{2}$).

6.4. Comparison to the Statistical Surveillance Baselines

Considering the results of the previous experiment, we have chosen to evaluate the $p$-value and the standard fusion approach with a window size of $1$, the adaption of the labeling $O_{3}$ and including the mean. In order to draw conclusions, we have evaluated the underlying statistical methods itself which serve as a baseline.

The results for the average rank are represented in Table 4. Here, we can observe that $p$-value fusion achieves the best rating across all test cases. In contrast, the performance of the standard fusion approach is often worse than the underlying statistical algorithms. In line with Texier et al. (2019) and Jafarpour et al. (2013), we can observe an improvement of the standard fusion approach on the time series without trend and seasonality. However, this improvement is not consistent for all compared test cases, resulting only in an average rank of $3.0$ while our proposed $p$-value approach always achieves the best result. Indeed, the ability to detect outbreaks with the standard fusion approach is reduced since it is based on the output of the statistical algorithms given a particular pre-defined significance level $\alpha$ for them. This limits the information about sudden changes encapsulated in the training data which makes it pretty difficult for the ML algorithm to identify valuable patterns. A closer examination reveals that trend and seasonality has an impact on the evaluated stacking approaches. In particular, by learning over the baseline data of time series which include trend, the learner is fed with observations which are not representative for the future (evaluation data) due to the changed circumstances. Moreover, the learning algorithm usually assumes that the instances are considered to be independent and identically distributed in the learning data set, not allowing to capture concept drift. Our proposed approaches are not designed to adjust to these settings but we believe that further investigations on the influence of trend and seasonality and how they can be handled is an interesting avenue for future work.

Furthermore, we have evaluated the approaches with respect to the number of cases per outbreak. In contrast to the previous experiments, where the value for the parameter $k$ (used to define the number of cases per outbreak) was randomly drawn between $1$ and $10$, we have fixed this parameter to a particular value for all time series of the $42$ test cases. The results for the measure $dAUC_{1\%}$ across the $42$ test cases with a fixed value for the parameter $k$ is visualized as box plots, representing minimum, first quantile, mean, third quantile and maximum, in Figure 3. In addition to $dAUC_{1\%}$, we include the analysis of the $pAUC_{1\%}$ measure and compare to the original labeling $O_{0}$ in order to further investigate the effect of the labeling on detection rate and true positive rate.

As the cases per outbreak increases all methods are more likely to obtain a better performance. While the C1, C2, C3 and RKI method achieve comparable results across all outbreak sizes, we are surprised to observe that the Bayes method has a better performance in case of larger outbreaks. This contradicts our expectation that the RKI method should obtain the best results across these methods since the Poisson assumption was specifically used to generate the synthetic data.

Regarding the $p$-value fusion approaches, the results confirm the better overall performance across all outbreak sizes while the performance of the standard fusion approach gets worse compared to the other methods with an increasing number of cases per outbreak. This gives further evidence that the standard fusion is not ideal. A closer examination of the graphs for the measures $dAUC_{1\%}$ and $pAUC_{1\%}$ reveals the difference between the adaption of the labeling for the learning. In particular, without adaption the ML algorithm achieves a tremendously better performance for the trade-off between the true positive rate and the false alarm rate. However, this also has an effect on the ability to detect outbreaks as discussed in Section 4.3, yielding a slightly worse result for the measure $dAUC_{1\%}$ than with adapting the labeling.

7. Conclusions

In this work, we introduced an approach for the fusion of outbreak detection methods using machine learning, more specifically stacking. The original idea is to use the alarm or no alarm prediction of the underlying statistical algorithms as inputs to the learner. We improved that setup by incorporating the $p$-values instead, which contain more information about the certainty of an event than the simple binary outputs. In addition, we proposed to add additional information to the learning data and to adapt the labeling of an outbreak in order to improve the ability to detect outbreaks. For evaluation, we proposed a measure based on ROC curves which better adapts to the specific need for a very low false alarm rate but still considers the trade-off with the detection rate.

Our experimental results on synthetic data show that the fusion of $p$-values improves the performance compared to the underlying statistical algorithms. Contrary to previous work, we could also observe that simple fusion of binary outputs using stacking does not always lead to an improvement. By incorporating additional information to the learning data, more specifically the mean count of the previous observations and the previous outputs of the statistical methods, the machine learning algorithm is able to capture more reliable patterns to detect outbreaks. Furthermore, the labeling of an outbreak has an influence on the performance for the classification algorithm to detect outbreaks. By setting the focus on the peak of an outbreak during the learning process, a better performance to detect sudden changes can be achieved.

The effectiveness of the proposed method has still to be confirmed on real data. Nevertheless, our results suggest that $p$-value stacking is generally well-suited for combining the outcomes of established methods for outbreak detection with only a low risk of decreasing performance. Moreover, stacking allows to enrich the detection by additional signals and sources of information in a highly flexible way. However, a major challenge remains the treatment of the outbreak annotations during training, since these labels are inherently non-binary (endemic vs. epidemic) and additionally noisy and unreliable for real data.

Acknowledgements.

This work was supported by the Innovation Committee of the Federal Joint Committee (G-BA) [ESEG project, grant number 01VSF17034]