Detection of Unknown Errors in Human-Centered Systems

TL;DR

This paper introduces a model-agnostic method using hybrid neural networks and conformal inference to detect unknown errors in human-centered AI systems, enhancing safety by identifying errors before harm occurs.

Contribution

It presents a novel approach combining physics-based modeling and conformal inference for early error detection without prior error signature knowledge.

Findings

Outperforms existing methods in detecting unknown errors

Effective across multiple real-world safety-critical systems

Enables early intervention before unsafe data shifts occur

Abstract

Artificial Intelligence-enabled systems are increasingly being deployed in real-world safety-critical settings involving human participants. It is vital to ensure the safety of such systems and stop the evolution of the system with error before causing harm to human participants. We propose a model-agnostic approach to detecting unknown errors in such human-centered systems without requiring any knowledge about the error signatures. Our approach employs dynamics-induced hybrid recurrent neural networks (DiH-RNN) for constructing physics-based models from operational data, coupled with conformal inference for assessing errors in the underlying model caused by violations of physical laws, thereby facilitating early detection of unknown errors before unsafe shifts in operational data distribution occur. We evaluate our framework on multiple real-world safety critical systems and show that our technique outperforms the existing state-of-the-art in detecting unknown errors.