Multiple Document Representations from News Alerts for Automated Bio-surveillance Event Detection

TL;DR

This paper introduces a method for automated bio-surveillance event detection using multiple document representations, achieving high accuracy and recall in classifying health-related news documents.

Contribution

It presents a novel approach combining recurrent neural networks and traditional text representations for improved disease event classification.

Findings

Achieved 97% recall and 93.3% accuracy in classification

Joint modeling of document title and description improves performance

Recurrent neural network-based representations outperform TF-IDF and Naive Bayes

Abstract

Due to globalization, geographic boundaries no longer serve as effective shields for the spread of infectious diseases. In order to aid bio-surveillance analysts in disease tracking, recent research has been devoted to developing information retrieval and analysis methods utilizing the vast corpora of publicly available documents on the internet. In this work, we present methods for the automated retrieval and classification of documents related to active public health events. We demonstrate classification performance on an auto-generated corpus, using recurrent neural network, TF-IDF, and Naive Bayes log count ratio document representations. By jointly modeling the title and description of a document, we achieve 97% recall and 93.3% accuracy with our best performing bio-surveillance event classification model: logistic regression on the combined output from a pair of bidirectional recurrent neural networks.

Figures

Tables

Table 1. Table 1: Statistics for assembled labeled and unlabeled corpora.

Corpus	Number of Texts	Vocabulary Size	Median length	Average Length
Unlabeled-Title	1,013,804	103,214	12	13
Unlabeled-Description	1,149,959	241,060	30	47
Labeled-Title	30,893	64,886	13	14
Labeled-Description	30,893	24,715	37	82

Table 2. Table 2: Recall, precision, F-measure, and accuracy for each model using the title and/or description as input.

Model	Text	Prec.	Rec.	F-scr.	Acc.
$D_{\text{TF}}$	Desc.	89.3	94.4	91.8	91.5
	Title.	88.9	94.2	91.5	91.2
	Both	91.0	95.1	93.0	92.8
	Desc.	90.1	93.2	91.6	91.4
$D_{\text{LCR}}$	Title	91.2	92.3	91.7	91.6
	Both	91.2	94.8	93.0	92.8
	Desc.	89.7	95.8	92.7	92.4
	Title	88.9	94.1	91.4	91.2
$D_{\text{RNN}}$	Both	90.4	97.0	93.6	93.3

Table 3. Table 3: Confusion matrix and metrics from analyst review of classifier.

	Event	Other			Precision	Recall
Event	\collectcell 60\endcollectcell	\collectcell 7\endcollectcell			\collectcell 0.73\endcollectcell	\collectcell 0.90\endcollectcell
Other	\collectcell 22\endcollectcell	\collectcell 23\endcollectcell

Table 4. Table 4: False negative title examples.

Tuberculosis epidemic affects the penalties of the Hopper,

the well and female jail

Meningitis

Outbreak of Hepatitis A in Sinaloa

Legionella found in Rendac water purification plant

Table 5. Table 5: False positive title examples.

They created an ecological trap to kill the

mosquitoes that produce Zika and Dengue

Study: 7% risk of birth defects in Zika pregnancies

Beginning of the end for Ghouta rebels:

Thousands flee relentless regime assault

PRO/AH/EDR$>$ Salmonellosis, st Agbeni -

USA: pet turtles

Equations

$$D_{\text{TF}}(d_k{)}_j=\frac{{\text{F}}_{k,j}}{\parallel {\text{F}}_{k,:}{\parallel }_{\infty }}\log \frac{N}{n_j}$$

$${\text{y}}_k$$

$$={\text{y}}^{⊺}(\text{1}+\text{F})\in {\mathbb{R}}^{1\times |V|}$$

$$=(\text{1 - y}{)}^{⊺}(\text{1}+\text{F})\in {\mathbb{R}}^{1\times |V|}$$

$$=\log \frac{\text{p}\backslash \parallel \text{p}{\parallel }_1}{\text{q}\backslash \parallel \text{q}{\parallel }_1}\in {\mathbb{R}}^{1\times |V|}$$

$$D_{\text{LCR}}(d_k)=\text{sign}({\text{F}}_{k,:})\odot \text{r}$$

$$D_{\text{RNN}}=\left[\begin{array}{ccc}\frac{1}{k}\ast {\sum }_{t=1}^k{\mathbf{h}}_t& {\mathbf{h}}_0^b& {\mathbf{h}}_{k+1}^f\end{array}\right],$$

$$\displaystyle D_{\texttt{*}}\bigl{(}d_{\texttt{desc}}\bigr{)},$$

$$\displaystyle D_{\texttt{*}}\bigl{(}d_{\texttt{title}}\bigr{)}\textrm{ and,}$$

$$\displaystyle\begin{bmatrix}D_{\texttt{*}}\bigl{(}d_{\texttt{desc}}\bigr{)}&D_{\texttt{*}}\bigl{(}d_{\texttt{title}}\bigr{)}\end{bmatrix}$$

Full text

1 Introduction

Automated mining and analysis of open source web-based content provides a powerful aid in effective monitoring of public health concerns Hartley et al. (2013); Volkova et al. (2017); Pavalanathan et al. (2016); Milinovich et al. (2014); Charles-Smith et al. (2015). A deluge of potentially relevant content can quickly be generated from web monitoring services; overwhelming the cognitive capabilities of human analysts tasked with identifying and assessing current biosurveillance events. This glut of information combined with the great cost associated with missing timely events of importance affirms the need for reliable downstream classification of retrieved documents.

To train a biosurveillance current event document classification system, we assembled an analyst labeled data set of 30,893 articles from Google Alerts111https://www.google.com/alerts Kataru (2016). The news alerts provide multiple views of a news article: a title and description. These multiple views are a distinctive feature of our data set; there are two natural language texts, a title and description, associated with each news article. This offers a direct avenue to compare the effectiveness of joint document representations for traditional bag-of-N-grams models against their more sophisticated neural network successors, which, to our knowledge, is a novel contribution. In this work, we explore joint models of multiple document representations for classification. We compare the effectiveness of joint modeling for both state-of-the-art Recurrent Neural Networks (RNNs) and traditional bag-of-N-grams models. We find that classification performance is enhanced more significantly by the joint modeling of multiple text representations than by a choice of document model between bag-of-N-grams models and RNN.

2 Related Work

Recurrent neural networks with LSTM Hochreiter and Schmidhuber (1997) cells are often fa- vored as document models for their potential to model long term word dependencies. Miyato et al. (2016) use bidirectional Schuster and Paliwal (1997) LSTMs with unsupervised pre-training and adversarial noise regularization on the initial embedding vectors. They concatenate the final hidden states of the forward and backward LSTM as the document representation for classification. Liu et al. (2015) use max pooling over the hidden states of a bidirectional LSTM for classification. In contrast to these works, we use average pooling over the hidden states concatenated with the final hidden state to represent the text as a vector.

Joint modeling using multiple representations has demonstrated reliable performance gains for several neural network document classification models. Limsopatham and Collier (2016) create separate document models from two sets of pre-trained word vectors using texts from the classification target domain and a large unlabeled corpus. They found vectorizing the word embedding matrices with separate convolutional networks performed better than using combined matrices as input to a single convolutional network. McCann et al. (2017) use GloVe Pennington et al. (2014) word embeddings in tandem with pre-trained word embeddings from the English-to-German translation task. Johnson and Zhang (2016) improve performance using joint bidirec- tional LSTM representations of a document; one network has weights fixed after unsupervised pre-training while the other is learned from labeled documents. They showed similar gains using this same semi-supervised set-up but with convolutional networks in place of LSTMs.

3 Methods

In this section we describe our methods for bio-surveillance text retrieval and classification.

3.1 Retrieval

We retrieve potentially relevant documents con- taining disease names, acronyms, and synonyms through Google Alerts service. The set of queries, created by biosurveillance analysts, return alerts sent to a dedicated electronic mailbox, which is periodically mined for new content. An example alert returned for the ebola query:

title

Ebola survivors sue state over disappearance of aid money

description

Two Ebola survivors are to sue the government of Sierra Leone in the first international court case aimed at throwing light on what happened to some of the millions of dollars siphoned off from funds intended to help fight the disease. The case, filed…

The alerts provide a title, description, and URL link to the full text. Table 1 records the data-set statistics. 30,893 title/description document pairs were hand labeled by analysts, of which 15,331 were considered events of interest, and 15,362 were considered irrelevant to bio-surveillance. The title/description of 30,893 documents were labeled at a near 50% split for the binary classification task of detecting active biosurveillance events. This data was labeled by 8 biosurveillance analysts and reviewed by a lead analyst. About 1M unlabeled alert texts were also retrieved for analysis.

3.2 Document Representation

We consider three vector representions of documents, term frequency inverse document frequency ($D_{\texttt{TF}}$), log-count ratio ($D_{\texttt{LCR}}$), and a bidirectional recurrent neural network document representation ($D_{\texttt{RNN}}$). Our vocabulary $V$ is the set of all bigrams and unigrams that appear in the training corpus of $N$ documents. The matrix $\textbf{F}\in\mathbb{R}^{N\times|V|}$ is the document count matrix such that $\textbf{F}_{k,j}$ is the number of times term $j$ occurs in document $k$. The $j$-th element of the $D_{\texttt{TF}}$ vector for document $d_{k}$ is then:

[TABLE]

where $n_{j}$ is the number of documents in the training corpus where term $j$ appears at least once, and $\parallel*\parallel_{\infty}$ is the max norm of a vector.

We use an add-one smoothed term-count matrix to derive a log-count ratio vector for all terms in the vocabulary. Let p and q be the smoothed positive and negative term count vectors. Then:

[TABLE]

where 1 is a suitably sized vector or matrix of ones, and $\frac{*}{*}$ is elementwise division. Now our vector representation $D_{\texttt{LCR}}$ is defined as:

[TABLE]

where $\odot$ is elementwise multiplication. The sign function binarizes F into a matrix of 1’s and 0’s.

For the recurrent neural network representation, $D_{\texttt{RNN}}$, we use the GloVe algorithm to create vectors in $\mathbb{R}^{256}$ representing each word in an enlarged unigram dictionary from the combined corpus of unlabeled documents and a training set of the labeled documents. This allows high quality vector representations of words absent in the training set.

As depicted in figure 1, we model documents from word vector sequences using a bidirectional LSTM. Let $\mathcal{X}={\bf x}_{1},{\bf x}_{2},...,{\bf x}_{k}$ be the sequence of GloVe vectors associated with a document comprised of $k$ tokens. Let ${\bf h}^{f}_{t}$ be the $t$-th hidden state of the forward LSTM and ${\bf h}^{b}_{t}$ the $(k-t)$-th hidden state of the backward LSTM. Consider the hidden states as row vectors, and let ${\bf h}_{t}=\begin{bmatrix}{\bf h}^{b}_{t}&{\bf h}^{f}_{t}\end{bmatrix}$. Our RNN document vector representation is:

[TABLE]

the means of the forward and backwards hidden states concatenated with the final LSTM outputs.

3.3 Classification

We fit logistic regression models with L2 regularization to classify documents for each of the three document representations $D_{\texttt{TF}}$, $D_{\texttt{LCR}}$, and $D_{\texttt{RNN}}$. For each news article, $d$, in our labeled corpus, we’ve obtained two views from the Google Alerts; $d_{\texttt{title}}$, the alert title, and $d_{\texttt{desc}}$, the alert description. For $D_{*}:*\in\{\texttt{TF},\texttt{LCR},\texttt{RNN}\}$ we consider feature vectors:

[TABLE]

which are inputs to a logistic regression (LR) classifier using the description, the title, or both the title and description. For the $D_{\texttt{RNN}}$ representation we jointly learn the RNN parameters and logistic regression parameters. For the $D_{\texttt{TF}}$, and $D_{\texttt{LCR}}$ models, separate term count matrices are calculated for the title and description texts.

4 Experiments

Our principal set of experiments is designed to test the classifier performance across two axes: the document model, and the text input. We explore nine LR model configurations; three document models (TF, LCR, RNN) each using either text only, description only, or both text and description. An additional set of experiments assesses performance of the $D_{\texttt{TF}}$ representations for a suite of three other classifiers, Naive Bayes, Support Vector Machine, and Random Forest.

4.1 Experiment Set-up

The joint RNN, logistic regression model is coded in Tensorflow Abadi et al. (2016). For the $D_{\texttt{LCR}}$ classification model, we adapt open source code222https://github.com/mesnilgr/nbsvm from Grégoire Mesnil which implements the logistic regression variant of NB-SVM Wang and Manning (2012). We implement the $D_{\texttt{TF}}$ classification model for logistic regression and several other classifiers using scikit-learn Pedregosa et al. (2011).

The labeled documents are split into train, de- velopment, and test sets with 24,715 documents in the train set, and 3,089 documents each in the development and test sets. A random search was performed over hyper-parameters for each model and results are reported on the test set for the best performing development set models.

4.2 Results and Analysis

Table 2 shows performance for nine logistic regression models. Our first observation is that both bag-of-N-grams models and the RNN perform better using a joint model of title and description. Further, the performance gain from using joint modeling is more significant than the choice of document vector representation for ac- curacy and F-measure. Since high recall is critical for the bio-surveillance application, the joint $D_{\texttt{RNN}}$ representation is most suitable with 97% recall, followed by the description $D_{\texttt{RNN}}$ which attains 95.8% recall. The other two $D_{\texttt{LCR}}$ and $D_{\texttt{TF}}$ joint representations also give good recall with 95.1% and 94.8% respectively. For the RNN representation, using the longer description text performs better than using the shorter title text. This speaks to the RNN representation’s capability to model long term dependencies in the text; a capability lacking in the simpler bag-of-Ngram models. Figure 6 confirms the advantage of a $D_{\texttt{TF}}$ joint representation over a wider range of models.

Figures 5, 5, and 5 plot t-SNE Maaten and Hinton (2008) visualizations of the description, title, and joint RNN representations. The 2D visualization suggests that title and description vectors are mapped onto different manifolds in high dimensional space. For most models, title and description representations perform similarly, suggesting that performance gains from joint modeling may be from regularizing effects of complementary representations.

4.3 Online Performance

In this section, we review online performance of the best performing active public health event classifier (the joint RNN document representation) as deployed in an application monitoring the live stream of online sources.

The performance results from analyst review of 122 document classification predictions are recorded in Table 3. High recall is of most importance due to the high cost of false negatives which could lead to overlooking active public health events. The model maintains a high recall with a slight drop when moving from our test set to live data, but shows a significant drop in precision due to a large number of false positives.

Tables 4 and 5 show examples of false positives and false negatives from the 122 reviewed online documents, respectively. The false positive examples present text which is similar to text which might describe an active public health event. The false negative examples are clearly about urgent events of interest: a tuberculosis epidemic, a meningitis case, a hepatitis outbreak, and a water source tainted with Legionella. These omissions could be due to lack of coverage for these diseases in the labeled training data or articles which mention past but not current outbreaks of these diseases which would be labeled as non-events by an analyst.

5 Conclusion

Here, we present simple and effective components for biosurveillance monitoring. Importantly, the tools for building a system are openly accessible, making event detection available to a broad range of interests. Our best performing method employs a pair of bidirectional recurrent neural networks, jointly modeling the text and description of a document with a logistic regression classifier (93.3% accuracy; 97% recall). We find that joint representation models achieve similar performance gains for both recurrent neural networks and traditional bag-of-N-grams models.

We are currently exploring a more general class of joint document representation models which this work does not address, e.g., combining bag-of-N-grams models with neural network models, combining log count ratio vectors with TF-IDF vectors, and other permutations. Finally, we note that, for other data sets, bag-of-N-grams models may also be enhanced through joint modeling. Traditional feature selection methods Yang and Pedersen (1997) can be used to generate multiple document views for joint models through document term thresholding.

Acknowledgments

This work was funded by Department of Homeland Security, Homeland Security Advanced Research Projects Agency-Chemical Biological Division under Agreement # HSHQPM-17-00077.