Diagnostic Runtime Monitoring with Martingales

TL;DR

This paper introduces a new streaming framework using multiple stochastic martingales to diagnose distribution shifts in machine learning systems, enabling timely interventions and improved robustness in safety-critical robotics.

Contribution

It presents a novel martingale-based approach for real-time diagnosis of distribution shifts, adaptable across various models and datasets, outperforming existing methods.

Findings

Outperforms existing methods in speed and accuracy

Effective in both simulated and real hardware environments

Flexible framework adaptable to different shift types

Abstract

Machine learning systems deployed in safety-critical robotics settings must be robust to distribution shifts. However, system designers must understand the cause of a distribution shift in order to implement the appropriate intervention or mitigation strategy and prevent system failure. In this paper, we present a novel framework for diagnosing distribution shifts in a streaming fashion by deploying multiple stochastic martingales simultaneously. We show that knowledge of the underlying cause of a distribution shift can lead to proper interventions over the lifecycle of a deployed system. Our experimental framework can easily be adapted to different types of distribution shifts, models, and datasets. We find that our method outperforms existing work on diagnosing distribution shifts in terms of speed, accuracy, and flexibility, and validate the efficiency of our model in both simulated and live hardware settings.