Bayes Watch: Bayesian Change-point Detection for Process Monitoring with Fault Detection

TL;DR

Bayes Watch is a Bayesian framework for real-time change-point detection in high-dimensional, mixed-type longitudinal data, enabling early identification of model performance issues and pinpointing responsible features.

Contribution

It introduces a novel Bayesian change-point detection method using Gaussian Graphical Mixture Models for high-dimensional data with missing values.

Findings

Effective detection of change-points in complex data.

Identifies variables responsible for detected changes.

Provides probabilistic assessment of change-point locations.

Abstract

When a predictive model is in production, it must be monitored in real-time to ensure that its performance does not suffer due to drift or abrupt changes to data. Ideally, this is done long before learning that the performance of the model itself has dropped by monitoring outcome data. In this paper we consider the problem of monitoring a predictive model that identifies the need for palliative care currently in production at the Mayo Clinic in Rochester, MN. We introduce a framework, called \textit{Bayes Watch}, for detecting change-points in high-dimensional longitudinal data with mixed variable types and missing values and for determining in which variables the change-point occurred. Bayes Watch fits an array of Gaussian Graphical Mixture Models to groupings of homogeneous data in time, called regimes, which are modeled as the observed states of a Markov process with unknown transition probabilities. In doing so, Bayes Watch defines a posterior distribution on a vector of regime assignments, which gives meaningful expressions on the probability of every possible change-point. Bayes Watch also allows for an effective and efficient fault detection system that assesses what features in the data where the most responsible for a given change-point.