Composable Models for Online Bayesian Analysis of Streaming Data

TL;DR

This paper introduces a flexible, composable framework using particle filters for real-time Bayesian analysis of irregular, streaming IoT sensor data, accommodating non-linear, continuous-time Markov processes.

Contribution

It presents a novel Scala library enabling modular modeling of continuous-time Markov processes for online IoT data analysis.

Findings

Developed a Scala library for composable continuous-time Markov models.

Demonstrated real-time analysis of irregular IoT sensor data.

Supported non-linear, partially observed processes in streaming environments.

Abstract

Data is rapidly increasing in volume and velocity and the Internet of Things (IoT) is one important source of this data. The IoT is a collection of connected devices (things) which are constantly recording data from their surroundings using on-board sensors. These devices can record and stream data to the cloud at a very high rate, leading to high storage and analysis costs. In order to ameliorate these costs, we can analyse the data as it arrives in a stream to learn about the underlying process, perform interpolation and smoothing and make forecasts and predictions. Conventional tools of state space modelling assume data on a fixed regular time grid. However, many sensors change their sampling frequency, sometimes adaptively, or get interrupted and re-started out of sync with the previous sampling grid, or just generate event data at irregular times. It is therefore desirable to model the system as a partially and irregularly observed Markov process which evolves in continuous time. Both the process and the observation model are potentially non-linear. Particle filters therefore represent the simplest approach to online analysis. A functional Scala library of composable continuous time Markov process models has been developed in order to model the wide variety of data captured in the IoT.