Evaluating Different Fault Injection Abstractions on the Assessment of DNN SW Hardening Strategies

TL;DR

This paper compares application-level and instruction-level fault injection methods to evaluate software hardening strategies for neural networks, revealing that the choice of abstraction significantly impacts reliability assessments.

Contribution

It is the first study to compare FI approaches at different abstraction levels for evaluating DNN SW hardening, emphasizing the importance of analysis granularity.

Findings

Instruction-level FI alters the ranking of hardening techniques.

Application-level FI may lead to misleading reliability assessments.

Choosing the appropriate FI abstraction is crucial for accurate evaluation.

Abstract

The reliability of Neural Networks has gained significant attention, prompting efforts to develop SW-based hardening techniques for safety-critical scenarios. However, evaluating hardening techniques using application-level fault injection (FI) strategies, which are commonly hardware-agnostic, may yield misleading results. This study for the first time compares two FI approaches (at the application level (APP) and instruction level (ISA)) to evaluate deep neural network SW hardening strategies. Results show that injecting permanent faults at ISA (a more detailed abstraction level than APP) changes completely the ranking of SW hardening techniques, in terms of both reliability and accuracy. These results highlight the relevance of using an adequate analysis abstraction for evaluating such techniques.