A framework for the automation of testing computer vision systems

TL;DR

This paper introduces a framework for automating the testing of computer vision systems by generating and modifying images to evaluate system reliability, with applications in industrial defect detection and image classification.

Contribution

It presents a novel framework that automates test generation for vision systems using image modification and similarity measures, addressing a gap in quality assurance methods.

Findings

Framework successfully tests vision systems in industrial defect detection.

Preliminary results demonstrate effectiveness in image classification tasks.

Utilizes existing libraries for image modification and similarity assessment.

Abstract

Vision systems, i.e., systems that allow to detect and track objects in images, have gained substantial importance over the past decades. They are used in quality assurance applications, e.g., for finding surface defects in products during manufacturing, surveillance, but also automated driving, requiring reliable behavior. Interestingly, there is only little work on quality assurance and especially testing of vision systems in general. In this paper, we contribute to the area of testing vision software, and present a framework for the automated generation of tests for systems based on vision and image recognition. The framework makes use of existing libraries allowing to modify original images and to obtain similarities between the original and modified images. We show how such a framework can be used for testing a particular industrial application on identifying defects on riblet surfaces and present preliminary results from the image classification domain.