Learning the Noise of Failure: Intelligent System Tests for Robots

TL;DR

This paper introduces a novel method using simulated airborne noise and machine learning to detect software failures in robots, enhancing automated testing accuracy and reducing reliance on time-consuming real-world experiments.

Contribution

The paper presents a new noise-based failure detection technique using support vector machines, applicable across different robots, and provides an open-source tool for broader adoption.

Findings

High failure detection accuracy with low false positives

Single model effective across multiple robot types

Open-source tool NoisyTest facilitates easy testing

Abstract

Roboticists usually test new control software in simulation environments before evaluating its functionality on real-world robots. Simulations reduce the risk of damaging the hardware and can significantly increase the development process's efficiency in the form of automated system tests. However, many flaws in the software remain undetected in simulation data, revealing their harmful effects on the system only in time-consuming experiments. In reality, such irregularities are often easily recognized solely by the robot's airborne noise during operation. We propose a simulated noise estimate for the detection of failures in automated system tests of robots. The classification of flaws uses classical machine learning - a support vector machine - to identify different failure classes from the scalar noise estimate. The methodology is evaluated on simulation data from the humanoid robot LOLA. The approach yields high failure detection accuracy with a low false-positive rate, enabling its use for stricter automated system tests. Results indicate that a single trained model may work for different robots. The proposed technique is provided to the community in the form of the open-source tool NoisyTest, making it easy to test data from any robot. In a broader scope, the technique may empower real-world automated system tests without human evaluation of success or failure.