How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy

TL;DR

This paper introduces a new approach for predicting safety in autonomous image-controlled systems using generative models and conformal calibration, addressing the lack of low-dimensional state representations.

Contribution

It proposes a configurable family of safety prediction pipelines that do not rely on low-dimensional states and provide statistical calibration guarantees.

Findings

Effective safety prediction in racing car and cartpole scenarios

Calibration guarantees improve reliability of safety assessments

Handles missing safety labels and distribution shifts

Abstract

End-to-end learning has emerged as a major paradigm for developing autonomous systems. Unfortunately, with its performance and convenience comes an even greater challenge of safety assurance. A key factor of this challenge is the absence of the notion of a low-dimensional and interpretable dynamical state, around which traditional assurance methods revolve. Focusing on the online safety prediction problem, this paper proposes a configurable family of learning pipelines based on generative world models, which do not require low-dimensional states. To implement these pipelines, we overcome the challenges of learning safety-informed latent representations and missing safety labels under prediction-induced distribution shift. These pipelines come with statistical calibration guarantees on their safety chance predictions based on conformal prediction. We perform an extensive evaluation of the proposed learning pipelines on two case studies of image-controlled systems: a racing car and a cartpole.