Revisiting Deep Neural Network Test Coverage from the Test Effectiveness Perspective

TL;DR

This study reevaluates the usefulness of various DNN test coverage metrics by examining their correlation with test effectiveness, revealing some positive insights and practical scenarios for their application.

Contribution

It provides a comprehensive validation of the fundamental assumption that DNN test coverage correlates with test effectiveness, highlighting their complementary roles and potential usage scenarios.

Findings

Structural and non-structural coverage are complementary.

Some test coverage metrics are useful in certain scenarios.

The study offers practical guidance for applying DNN test coverage.

Abstract

Many test coverage metrics have been proposed to measure the Deep Neural Network (DNN) testing effectiveness, including structural coverage and non-structural coverage. These test coverage metrics are proposed based on the fundamental assumption: they are correlated with test effectiveness. However, the fundamental assumption is still not validated sufficiently and reasonably, which brings question on the usefulness of DNN test coverage. This paper conducted a revisiting study on the existing DNN test coverage from the test effectiveness perspective, to effectively validate the fundamental assumption. Here, we carefully considered the diversity of subjects, three test effectiveness criteria, and both typical and state-of-the-art test coverage metrics. Different from all the existing studies that deliver negative conclusions on the usefulness of existing DNN test coverage, we identified some positive conclusions on their usefulness from the test effectiveness perspective. In particular, we found the complementary relationship between structural and non-structural coverage and identified the practical usage scenarios and promising research directions for these existing test coverage metrics.