Repeatable and Reliable Efforts of Accelerated Risk Assessment in Robot Testing

TL;DR

This paper introduces a new algorithm to improve the repeatability and reliability of accelerated robot risk assessments, ensuring consistent and fair safety evaluations across diverse testing conditions and robot types.

Contribution

The study proposes a novel algorithm that guarantees repeatability and reliability in accelerated risk assessments, addressing limitations of traditional Monte-Carlo and importance sampling methods.

Findings

Demonstrated risk assessment of robot stability under various control algorithms.

Validated the algorithm on inverted pendulum and bipedal robot tests.

Showed improved consistency and fairness in risk evaluation results.

Abstract

Risk assessment of a robot in controlled environments, such as laboratories and proving grounds, is a common means to assess, certify, validate, verify, and characterize the robots' safety performance before, during, and even after their commercialization in the real-world. A standard testing program that acquires the risk estimate is expected to be (i) repeatable, such that it obtains similar risk assessments of the same testing subject among multiple trials or attempts with the similar testing effort by different stakeholders, and (ii) reliable against a variety of testing subjects produced by different vendors and manufacturers. Both repeatability and reliability are fundamental and crucial for a testing algorithm's validity, fairness, and practical feasibility, especially for standardization. However, these properties are rarely satisfied or ensured, especially as the subject robots become more complex, uncertain, and varied. This issue was present in traditional risk assessments through Monte-Carlo sampling, and remains a bottleneck for the recent accelerated risk assessment methods, primarily those using importance sampling. This study aims to enhance existing accelerated testing frameworks by proposing a new algorithm that provably integrates repeatability and reliability with the already established formality and efficiency. It also features demonstrations assessing the risk of instability from frontal impacts, initiated by push-over disturbances on a controlled inverted pendulum and a 7-DoF planar bipedal robot Rabbit managed by various control algorithms.