Assessing the Uncertainty and Robustness of the Laptop Refurbishing Software

TL;DR

This paper evaluates the uncertainty and robustness of various sticker detection models used in laptop refurbishing software, employing Monte Carlo Dropout and adversarial datasets to improve reliability and safety.

Contribution

It introduces a novel framework for quantifying uncertainty and robustness of object detection models in laptop refurbishing, with new metrics and evaluation methods.

Findings

Different models show varied performance across metrics.

Uncertainty quantification helps in reducing risks during sticker removal.

Guidelines for selecting suitable detection models are provided.

Abstract

Refurbishing laptops extends their lives while contributing to reducing electronic waste, which promotes building a sustainable future. To this end, the Danish Technological Institute (DTI) focuses on the research and development of several robotic applications empowered with software, including laptop refurbishing. Cleaning represents a major step in refurbishing and involves identifying and removing stickers from laptop surfaces. Software plays a crucial role in the cleaning process. For instance, the software integrates various object detection models to identify and remove stickers from laptops automatically. However, given the diversity in types of stickers (e.g., shapes, colors, locations), identification of the stickers is highly uncertain, thereby requiring explicit quantification of uncertainty associated with the identified stickers. Such uncertainty quantification can help reduce risks in removing stickers, which, for example, could otherwise result in software faults damaging laptop surfaces. For uncertainty quantification, we adopted the Monte Carlo Dropout method to evaluate six sticker detection models (SDMs) from DTI using three datasets: the original image dataset from DTI and two datasets generated with vision language models, i.e., DALL-E-3 and Stable Diffusion-3. In addition, we presented novel robustness metrics concerning detection accuracy and uncertainty to assess the robustness of the SDMs based on adversarial datasets generated from the three datasets using a dense adversary method. Our evaluation results show that different SDMs perform differently regarding different metrics. Based on the results, we provide SDM selection guidelines and lessons learned from various perspectives.