Model-based Test Generation for Robotic Software: Automata versus Belief-Desire-Intention Agents

TL;DR

This paper compares automata-based and BDI agent-based model-driven test generation methods for robotic software, demonstrating that BDI agents offer more expressiveness and comparable or better coverage and performance in complex interaction scenarios.

Contribution

It introduces and evaluates BDI agents as a novel, more expressive alternative to automata for model-based test generation in robotic software testing.

Findings

BDI agents are more expressive than automata for modeling human-robot interactions.

BDI-based test generation achieves higher or comparable coverage.

Performance of BDI-based testing is at least as good as automata-based methods.

Abstract

Robotic code needs to be verified to ensure its safety and functional correctness, especially when the robot is interacting with people. Testing real code in simulation is a viable option. However, generating tests that cover rare scenarios, as well as exercising most of the code, is a challenge amplified by the complexity of the interactions between the environment and the software. Model-based test generation methods can automate otherwise manual processes and facilitate reaching rare scenarios during testing. In this paper, we compare using Belief-Desire-Intention (BDI) agents as models for test generation with more conventional automata-based techniques that exploit model checking, in terms of practicality, performance, transferability to different scenarios, and exploration (`coverage'), through two case studies: a cooperative manufacturing task, and a home care scenario. The results highlight the advantages of using BDI agents for test generation. BDI agents naturally emulate the agency present in Human-Robot Interactions (HRIs), and are thus more expressive than automata. The performance of the BDI-based test generation is at least as high, and the achieved coverage is higher or equivalent, compared to test generation based on model checking automata.