Sequential Event Detection Using Multimodal Data in Nonstationary Environments

TL;DR

This paper presents a novel framework for detecting anomalies in nonstationary multimodal data by transforming sensor and social media data into count statistics and applying POMDP-based quickest detection methods.

Contribution

It introduces a new approach combining neural network-based data transformation with POMDP for anomaly detection in nonstationary, multimodal environments.

Findings

Effective detection of events in real-world NYC data

Handles nonstationary count data across multiple modalities

Provides structural results for optimal detection policies

Abstract

The problem of sequential detection of anomalies in multimodal data is considered. The objective is to observe physical sensor data from CCTV cameras, and social media data from Twitter and Instagram to detect anomalous behaviors or events. Data from each modality is transformed to discrete time count data by using an artificial neural network to obtain counts of objects in CCTV images and by counting the number of tweets or Instagram posts in a geographical area. The anomaly detection problem is then formulated as a problem of quickest detection of changes in count statistics. The quickest detection problem is then solved using the framework of partially observable Markov decision processes (POMDP), and structural results on the optimal policy are obtained. The resulting optimal policy is then applied to real multimodal data collected from New York City around a 5K race to detect the race. The count data both before and after the change is found to be nonstationary in nature. The proposed mathematical approach to this problem provides a framework for event detection in such nonstationary environments and across multiple data modalities.