Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance

TL;DR

This paper investigates how Winograd convolution can enhance neural network fault tolerance, reducing overhead and improving robustness against soft errors without sacrificing accuracy.

Contribution

It provides a comprehensive evaluation of Winograd convolution's fault tolerance and demonstrates its potential for cost-effective neural network protection.

Findings

Winograd convolution reduces fault-tolerant design overhead by 55.77%.

It further decreases computing overhead by 17.24% considering inherent fault tolerance.

Models with Winograd convolution show improved accuracy under fault conditions.

Abstract

Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution in improving neural network (NN) fault tolerance. Based on the observation, we evaluate winograd convolution fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of winograd convolution for cost-effective NN protection against soft errors. Specifically, we mainly investigate how winograd convolution can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), fault-aware retraining, and constrained activation functions. According to our experiments, winograd convolution can reduce the fault-tolerant design overhead by 55.77\% on average without any accuracy loss compared to standard convolution, and further reduce the computing overhead by 17.24\% when the inherent fault tolerance of winograd convolution is considered. When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.