Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Driving

TL;DR

This paper explores using pre-trained vision models as perception backbones to design safety filters for autonomous driving, demonstrating their effectiveness in high-dimensional vision-based control scenarios.

Contribution

It introduces a novel approach of leveraging frozen pre-trained vision models for safety filtering in autonomous driving, addressing high-dimensional perception challenges.

Findings

Pre-trained vision models can effectively serve as perception backbones for safety filters.

Safety filters based on these models perform competitively with ground truth-based filters.

The approach improves safety in vision-based autonomous driving scenarios.

Abstract

End-to-end vision-based autonomous driving has achieved impressive success, but safety remains a major concern. The safe control problem has been addressed in low-dimensional settings using safety filters, e.g., those based on control barrier functions. Designing safety filters for vision-based controllers in the high-dimensional settings of autonomous driving can similarly alleviate the safety problem, but is significantly more challenging. In this paper, we address this challenge by using frozen pre-trained vision representation models as perception backbones to design vision-based safety filters, inspired by these models' success as backbones of robotic control policies. We empirically evaluate the offline performance of four common pre-trained vision models in this context. We try three existing methods for training safety filters for black-box dynamics, as the dynamics over representation spaces are not known. We use the DeepAccident dataset that consists of action-annotated videos from multiple cameras on vehicles in CARLA simulating real accident scenarios. Our results show that the filters resulting from our approach are competitive with the ones that are given the ground truth state of the ego vehicle and its environment.