FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions

TL;DR

FitAct introduces a low-cost, post-training method to improve the error resilience of deep neural networks by bounding neuron activations, preventing fault propagation without extensive retraining.

Contribution

It proposes a novel fine-grained, post-trainable activation function approach that enhances DNN fault tolerance while maintaining accuracy and low overhead.

Findings

Outperforms state-of-the-art methods like Clip-Act and Ranger

Effective across models like AlexNet, VGG16, ResNet50

Adds manageable runtime and memory overheads

Abstract

Deep neural networks (DNNs) are increasingly being deployed in safety-critical systems such as personal healthcare devices and self-driving cars. In such DNN-based systems, error resilience is a top priority since faults in DNN inference could lead to mispredictions and safety hazards. For latency-critical DNN inference on resource-constrained edge devices, it is nontrivial to apply conventional redundancy-based fault tolerance techniques. In this paper, we propose FitAct, a low-cost approach to enhance the error resilience of DNNs by deploying fine-grained post-trainable activation functions. The main idea is to precisely bound the activation value of each individual neuron via neuron-wise bounded activation functions so that it could prevent fault propagation in the network. To avoid complex DNN model re-training, we propose to decouple the accuracy training and resilience training and develop a lightweight post-training phase to learn these activation functions with precise bound values. Experimental results on widely used DNN models such as AlexNet, VGG16, and ResNet50 demonstrate that FitAct outperforms state-of-the-art studies such as Clip-Act and Ranger in enhancing the DNN error resilience for a wide range of fault rates while adding manageable runtime and memory space overheads.