Property-Based Testing in Simulation for Verifying Robot Action Execution in Tabletop Manipulation

TL;DR

This paper presents a property-based testing framework using simulation to verify robot action correctness in tabletop manipulation, enhancing reliability especially for learning-based robot systems.

Contribution

It introduces a novel approach combining property-based testing with simulation for verifying robot behaviors, specifically encoding properties in an ontology and testing in Gazebo.

Findings

Framework accurately detects failed actions in simulated scenarios.

It provides detailed insights into failure types and locations.

Effective for verifying robot actions in varied tabletop tasks.

Abstract

An important prerequisite for the reliability and robustness of a service robot is ensuring the robot's correct behavior when it performs various tasks of interest. Extensive testing is one established approach for ensuring behavioural correctness; this becomes even more important with the integration of learning-based methods into robot software architectures, as there are often no theoretical guarantees about the performance of such methods in varying scenarios. In this paper, we aim towards evaluating the correctness of robot behaviors in tabletop manipulation through automatic generation of simulated test scenarios in which a robot assesses its performance using property-based testing. In particular, key properties of interest for various robot actions are encoded in an action ontology and are then verified and validated within a simulated environment. We evaluate our framework with a Toyota Human Support Robot (HSR) which is tested in a Gazebo simulation. We show that our framework can correctly and consistently identify various failed actions in a variety of randomised tabletop manipulation scenarios, in addition to providing deeper insights into the type and location of failures for each designed property.