Towards Quality Assurance of Software Product Lines with Adversarial Configurations

TL;DR

This paper explores the use of adversarial machine learning techniques to identify vulnerabilities in ML-based quality assurance methods for software product lines, revealing potential risks and limitations.

Contribution

It introduces adversarial ML attacks to expose weaknesses in SPL quality assurance models, highlighting the need for more robust testing approaches.

Findings

Adversarial configurations can cause up to 100% misclassification.

Accuracy drops by up to 5% due to adversarial attacks.

Implications for improving SPL quality assurance methods.

Abstract

Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning techniques are increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.