How Do We Fail? Stress Testing Perception in Autonomous Vehicles

TL;DR

This paper introduces a reinforcement learning-based method to identify likely failures in LiDAR perception systems of autonomous vehicles under adverse weather, aiding the development of more robust perception algorithms.

Contribution

We propose a novel reinforcement learning approach combined with physics-based data augmentation to efficiently find failure cases in LiDAR perception under adverse weather conditions.

Findings

Our method detects high-likelihood failures with minimal disturbances.

It outperforms baseline approaches in identifying failures.

The approach is computationally efficient across diverse scenarios.

Abstract

Autonomous vehicles (AVs) rely on environment perception and behavior prediction to reason about agents in their surroundings. These perception systems must be robust to adverse weather such as rain, fog, and snow. However, validation of these systems is challenging due to their complexity and dependence on observation histories. This paper presents a method for characterizing failures of LiDAR-based perception systems for AVs in adverse weather conditions. We develop a methodology based in reinforcement learning to find likely failures in object tracking and trajectory prediction due to sequences of disturbances. We apply disturbances using a physics-based data augmentation technique for simulating LiDAR point clouds in adverse weather conditions. Experiments performed across a wide range of driving scenarios from a real-world driving dataset show that our proposed approach finds high likelihood failures with smaller input disturbances compared to baselines while remaining computationally tractable. Identified failures can inform future development of robust perception systems for AVs.