Conformal Safety Shielding for Imperfect-Perception Agents

TL;DR

This paper introduces a conformal prediction-based safety shield for autonomous agents with imperfect perception, ensuring run-time safety guarantees by restricting actions based on perception uncertainty.

Contribution

It presents a novel shield construction that provides probabilistic safety guarantees for agents using learned perception components, applicable in high-dimensional observation scenarios.

Findings

Guarantees safety with high probability under perception errors.

Successfully applied to autonomous airplane taxiing system.

Provides a method to incorporate perception uncertainty into safety assurances.

Abstract

We consider the problem of safe control in discrete autonomous agents that use learned components for imperfect perception (or more generally, state estimation) from high-dimensional observations. We propose a shield construction that provides run-time safety guarantees under perception errors by restricting the actions available to an agent, modeled as a Markov decision process, as a function of the state estimates. Our construction uses conformal prediction for the perception component, which guarantees that for each observation, the predicted set of estimates includes the actual state with a user-specified probability. The shield allows an action only if it is allowed for all the estimates in the predicted set, resulting in local safety. We also articulate and prove a global safety property of existing shield constructions for perfect-perception agents bounding the probability of reaching unsafe states if the agent always chooses actions prescribed by the shield. We illustrate our approach with a case-study of an experimental autonomous system that guides airplanes on taxiways using high-dimensional perception DNNs.