On Safety Testing, Validation, and Characterization with Scenario-Sampling: A Case Study of Legged Robots

TL;DR

This paper introduces a scenario sampling framework to evaluate and compare the safety performance of various legged robot controllers in complex, uncertain environments, providing new insights into their safety characteristics.

Contribution

It proposes a novel scenario sampling-based testing framework for safety assessment of legged robots, addressing limitations of existing methods and enabling comprehensive safety characterization.

Findings

Effectively captures safety performance across diverse scenarios

Identifies safety insights not reported in prior studies

Enables comparison of different robot control methods

Abstract

The dynamic response of the legged robot locomotion is non-Lipschitz and can be stochastic due to environmental uncertainties. To test, validate, and characterize the safety performance of legged robots, existing solutions on observed and inferred risk can be incomplete and sampling inefficient. Some formal verification methods suffer from the model precision and other surrogate assumptions. In this paper, we propose a scenario sampling based testing framework that characterizes the overall safety performance of a legged robot by specifying (i) where (in terms of a set of states) the robot is potentially safe, and (ii) how safe the robot is within the specified set. The framework can also help certify the commercial deployment of the legged robot in real-world environment along with human and compare safety performance among legged robots with different mechanical structures and dynamic properties. The proposed framework is further deployed to evaluate a group of state-of-the-art legged robot locomotion controllers from various model-based, deep neural network involved, and reinforcement learning based methods in the literature. Among a series of intended work domains of the studied legged robots (e.g. tracking speed on sloped surface, with abrupt changes on demanded velocity, and against adversarial push-over disturbances), we show that the method can adequately capture the overall safety characterization and the subtle performance insights. Many of the observed safety outcomes, to the best of our knowledge, have never been reported by the existing work in the legged robot literature.