Automated Behaviour-Driven Acceptance Testing of Robotic Systems

TL;DR

This paper extends behaviour-driven development (BDD) to robotic systems, enabling automated specification, generation, and execution of acceptance tests to improve reliability and systematic validation of robotic applications.

Contribution

It introduces a domain-specific language and a knowledge graph-based approach for defining and verifying acceptance criteria in robotics, integrating with simulation tools for automated testing.

Findings

Automated acceptance testing reveals behavior variations in robotic pick-and-place tasks.

The approach improves systematic validation and fault detection in robotic applications.

Integration with simulation enables scalable and repeatable testing processes.

Abstract

The specification and validation of robotics applications require bridging the gap between formulating requirements and systematic testing. This often involves manual and error-prone tasks that become more complex as requirements, design, and implementation evolve. To address this challenge systematically, we propose extending behaviour-driven development (BDD) to define and verify acceptance criteria for robotic systems. In this context, we use domain-specific modelling and represent composable BDD models as knowledge graphs for robust querying and manipulation, facilitating the generation of executable testing models. A domain-specific language helps to efficiently specify robotic acceptance criteria. We explore the potential for automated generation and execution of acceptance tests through a software architecture that integrates a BDD framework, Isaac Sim, and model transformations, focusing on acceptance criteria for pick-and-place applications. We tested this architecture with an existing pick-and-place implementation and evaluated the execution results, which shows how this application behaves and fails differently when tested against variations of the agent and environment. This research advances the rigorous and automated evaluation of robotic systems, contributing to their reliability and trustworthiness.