Search-based Robustness Testing of Laptop Refurbishing Robotic Software

TL;DR

This paper introduces PROBE, a search-based method using multi-objective optimization to find minimal input perturbations that reveal failures in object detection models used in laptop refurbishing robots, improving robustness testing.

Contribution

The paper presents PROBE, a novel search-based robustness testing approach employing NSGA-II to efficiently identify failure-inducing perturbations in robotic object detection models.

Findings

PROBE is 3 to 7 times more effective than random search in finding failures.

Generated perturbations transfer across different models.

Metamorphic relations help assess model stability even without failures.

Abstract

The Danish Technological Institute (DTI) focuses on transferring advanced technologies (including robots) to the industry and the public sector. One key application is laptop refurbishment using specialized robots, aimed at promoting reuse, reducing electronic waste, and supporting the European Circular Economy Action Plan. The software of such robots often includes features that use object detection models to detect objects for various purposes, such as identifying screws for laptop disassembly or detecting stickers to remove them. Ensuring the robustness of such models to small input variations remains a critical challenge, and addressing it is important to avoid potential damage to laptops during refurbishment. In this paper, we propose PROBE, a search-based robustness testing approach that leverages multi-objective optimization to identify minimal, localized perturbations that expose failures in object detection models used in the software of laptop refurbishing robots. PROBE employs NSGA-II to systematically explore the perturbation space, optimizing for failure induction considering both localization and confidence, and perturbation magnitude, while enabling the discovery of diverse failure cases. Results show that PROBE is 3$\times$ to 7$\times$ more effective than random search in generating failure-inducing perturbations, while requiring smaller perturbation magnitudes, and that the generated perturbations transfer across models. We further show that metamorphic relations provide additional insights into model robustness, enabling the assessment of stability even in non-failing cases.