FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation

TL;DR

This paper analyzes the fault tolerance of deep neural networks in safety-critical applications and introduces a novel clipping-based activation method that significantly enhances their resilience to hardware faults.

Contribution

It proposes a new fault mitigation technique using clipped activation functions and systematically determines optimal clipping values to improve DNN fault tolerance.

Findings

Achieves 68.92% average accuracy improvement at 1e-5 fault rate

Demonstrates effectiveness on AlexNet and VGG-16 models trained on CIFAR-10

Provides a systematic approach to define clipping values for enhanced resilience

Abstract

Deep Neural Networks (DNNs) are widely being adopted for safety-critical applications, e.g., healthcare and autonomous driving. Inherently, they are considered to be highly error-tolerant. However, recent studies have shown that hardware faults that impact the parameters of a DNN (e.g., weights) can have drastic impacts on its classification accuracy. In this paper, we perform a comprehensive error resilience analysis of DNNs subjected to hardware faults (e.g., permanent faults) in the weight memory. The outcome of this analysis is leveraged to propose a novel error mitigation technique which squashes the high-intensity faulty activation values to alleviate their impact. We achieve this by replacing the unbounded activation functions with their clipped versions. We also present a method to systematically define the clipping values of the activation functions that result in increased resilience of the networks against faults. We evaluate our technique on the AlexNet and the VGG-16 DNNs trained for the CIFAR-10 dataset. The experimental results show that our mitigation technique significantly improves the resilience of the DNNs to faults. For example, the proposed technique offers on average 68.92% improvement in the classification accuracy of resilience-optimized VGG-16 model at 1e-5 fault rate, when compared to the base network without any fault mitigation.