Error Analysis and Improving the Accuracy of Winograd Convolution for Deep Neural Networks

TL;DR

This paper analyzes the floating point error in Winograd convolution algorithms used in deep neural networks, proves error bounds, and proposes methods like Huffman coding and mixed-precision to reduce errors and improve accuracy.

Contribution

It provides a detailed error analysis of Winograd algorithms, introduces new techniques to reduce FP errors, and demonstrates improved accuracy enabling larger block sizes.

Findings

Error bounds grow exponentially with convolution size

Modified algorithms have smaller error bounds

Proposed methods significantly reduce FP error

Abstract

Popular deep neural networks (DNNs) spend the majority of their execution time computing convolutions. The Winograd family of algorithms can greatly reduce the number of arithmetic operations required and is present in many DNN software frameworks. However, the performance gain is at the expense of a reduction in floating point (FP) numerical accuracy. In this paper, we analyse the worst case FP error and prove the estimation of norm and conditioning of the algorithm. We show that the bound grows exponentially with the size of the convolution, but the error bound of the \textit{modified} algorithm is smaller than the original one. We propose several methods for reducing FP error. We propose a canonical evaluation ordering based on Huffman coding that reduces summation error. We study the selection of sampling "points" experimentally and find empirically good points for the most important sizes. We identify the main factors associated with good points. In addition, we explore other methods to reduce FP error, including mixed-precision convolution, and pairwise summation across DNN channels. Using our methods we can significantly reduce FP error for a given block size, which allows larger block sizes and reduced computation.