ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

TL;DR

This paper introduces ProAct, a progressive training method for hybrid clipped activation functions that improve DNN resilience to hardware faults by optimizing thresholds layer-by-layer, reducing overhead and enhancing fault tolerance.

Contribution

The paper proposes a novel hybrid clipped activation function with progressive training to optimize thresholds, reducing memory overhead and improving fault resilience in DNNs.

Findings

Hybrid activation functions with progressive threshold training improve fault tolerance.

Neuron-wise clipping is only applied in the last layer for efficiency.

ProAct reduces training time compared to heuristic fault injection methods.

Abstract

Deep Neural Networks (DNNs) are extensively employed in safety-critical applications where ensuring hardware reliability is a primary concern. To enhance the reliability of DNNs against hardware faults, activation restriction techniques significantly mitigate the fault effects at the DNN structure level, irrespective of accelerator architectures. State-of-the-art methods offer either neuron-wise or layer-wise clipping activation functions. They attempt to determine optimal clipping thresholds using heuristic and learning-based approaches. Layer-wise clipped activation functions cannot preserve DNNs resilience at high bit error rates. On the other hand, neuron-wise clipping activation functions introduce considerable memory overhead due to the addition of parameters, which increases their vulnerability to faults. Moreover, the heuristic-based optimization approach demands numerous fault injections during the search process, resulting in time-consuming threshold identification. On the other hand, learning-based techniques that train thresholds for entire layers concurrently often yield sub-optimal results. In this work, first, we demonstrate that it is not essential to incorporate neuron-wise activation functions throughout all layers in DNNs. Then, we propose a hybrid clipped activation function that integrates neuron-wise and layer-wise methods that apply neuron-wise clipping only in the last layer of DNNs. Additionally, to attain optimal thresholds in the clipping activation function, we introduce ProAct, a progressive training methodology. This approach iteratively trains the thresholds on a layer-by-layer basis, aiming to obtain optimal threshold values in each layer separately.