Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures

TL;DR

This paper introduces an uncertainty-aware latent safety filter that uses epistemic uncertainty and conformal prediction to detect and prevent both known and unseen safety hazards in vision-based robotic control, enhancing safety in complex environments.

Contribution

The paper presents a novel safety filter leveraging epistemic uncertainty and conformal prediction to detect out-of-distribution hazards in latent space, improving safety guarantees for robotic systems.

Findings

Successfully detects unseen hazards using epistemic uncertainty.

Prevents safety violations in vision-based control tasks.

Effective in both simulation and real-world experiments.

Abstract

Recent advances in generative world models have enabled classical safe control methods, such as Hamilton-Jacobi (HJ) reachability, to generalize to complex robotic systems operating directly from high-dimensional sensor observations. However, obtaining comprehensive coverage of all safety-critical scenarios during world model training is extremely challenging. As a result, latent safety filters built on top of these models may miss novel hazards and even fail to prevent known ones, overconfidently misclassifying risky out-of-distribution (OOD) situations as safe. To address this, we introduce an uncertainty-aware latent safety filter that proactively steers robots away from both known and unseen failures. Our key idea is to use the world model's epistemic uncertainty as a proxy for identifying unseen potential hazards. We propose a principled method to detect OOD world model predictions by calibrating an uncertainty threshold via conformal prediction. By performing reachability analysis in an augmented state space-spanning both the latent representation and the epistemic uncertainty-we synthesize a latent safety filter that can reliably safeguard arbitrary policies from both known and unseen safety hazards. In simulation and hardware experiments on vision-based control tasks with a Franka manipulator, we show that our uncertainty-aware safety filter preemptively detects potential unsafe scenarios and reliably proposes safe, in-distribution actions. Video results can be found on the project website at https://cmu-intentlab.github.io/UNISafe