Assessing Systematic Weaknesses of DNNs using Counterfactuals

TL;DR

This paper introduces a computationally efficient method inspired by counterfactual explanations to validate whether specific semantic features cause systematic weaknesses in DNNs, especially in safety-critical applications like autonomous driving.

Contribution

It proposes a novel, cost-effective algorithm to attribute performance drops to semantic features, addressing the challenge of validating systematic weaknesses in DNNs.

Findings

Performance differences among pedestrian assets were observed.

Only some asset types caused performance reduction.

The method effectively distinguishes true semantic causes from confounding factors.

Abstract

With the advancement of DNNs into safety-critical applications, testing approaches for such models have gained more attention. A current direction is the search for and identification of systematic weaknesses that put safety assumptions based on average performance values at risk. Such weaknesses can take on the form of (semantically coherent) subsets or areas in the input space where a DNN performs systematically worse than its expected average. However, it is non-trivial to attribute the reason for such observed low performances to the specific semantic features that describe the subset. For instance, inhomogeneities within the data w.r.t. other (non-considered) attributes might distort results. However, taking into account all (available) attributes and their interaction is often computationally highly expensive. Inspired by counterfactual explanations, we propose an effective and computationally cheap algorithm to validate the semantic attribution of existing subsets, i.e., to check whether the identified attribute is likely to have caused the degraded performance. We demonstrate this approach on an example from the autonomous driving domain using highly annotated simulated data, where we show for a semantic segmentation model that (i) performance differences among the different pedestrian assets exist, but (ii) only in some cases is the asset type itself the reason for this reduction in the performance.