FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection

TL;DR

This paper introduces FAST, a feature selection-based method that enhances uncertainty-based test prioritization for neural networks by reducing noise from irrelevant features, thereby improving fault detection and model generalization.

Contribution

FAST is a novel feature selection approach that boosts existing test prioritization methods for DNNs by dynamically pruning noisy features during inference.

Findings

Improves APFD scores for test prioritization

Enhances distinction between high-confidence errors and correct predictions

Demonstrates scalability across diverse models and datasets

Abstract

Due to the vast testing space, the increasing demand for effective and efficient testing of deep neural networks (DNNs) has led to the development of various DNN test case prioritization techniques. However, the fact that DNNs can deliver high-confidence predictions for incorrectly predicted examples, known as the over-confidence problem, causes these methods to fail to reveal high-confidence errors. To address this limitation, in this work, we propose FAST, a method that boosts existing prioritization methods through guided FeAture SelecTion. FAST is based on the insight that certain features may introduce noise that affects the model's output confidence, thereby contributing to high-confidence errors. It quantifies the importance of each feature for the model's correct predictions, and then dynamically prunes the information from the noisy features during inference to derive a new probability vector for the uncertainty estimation. With the help of FAST, the high-confidence errors and correctly classified examples become more distinguishable, resulting in higher APFD (Average Percentage of Fault Detection) values for test prioritization, and higher generalization ability for model enhancement. We conduct extensive experiments to evaluate FAST across a diverse set of model structures on multiple benchmark datasets to validate the effectiveness, efficiency, and scalability of FAST compared to the state-of-the-art prioritization techniques.