Before Autonomy Takes Control: Software Testing in Robotics

TL;DR

This paper reviews the state-of-the-art in robotics software testing, highlighting unique challenges due to hardware interaction, environmental uncertainty, and autonomy, and maps existing research to software testing theory.

Contribution

It provides a comprehensive mapping study of 247 robotics testing papers, relating them to software testing theory and identifying key challenges and open questions.

Findings

Robotics testing is complex due to hardware and environment interactions.

Current research addresses various testing challenges but gaps remain.

The study highlights open questions for future research.

Abstract

Robotic systems are complex and safety-critical software systems. As such, they need to be tested thoroughly. Unfortunately, robot software is intrinsically hard to test compared to traditional software, mainly since the software needs to closely interact with hardware, account for uncertainty in its operational environment, handle disturbances, and act highly autonomously. However, given the large space in which robots operate, anticipating possible failures when designing tests is challenging. This paper presents a mapping study by considering robotics testing papers and relating them to the software testing theory. We consider 247 robotics testing papers and map them to software testing, discussing the state-of-the-art software testing in robotics with an illustrated example, and discuss current challenges. Forming the basis to introduce both the robotics and software engineering communities to software testing challenges. Finally, we identify open questions and lessons learned.