Anderson-type acceleration method for Deep Neural Network optimization

TL;DR

This paper introduces an Anderson-type acceleration method to enhance stochastic gradient descent for deep neural network training, significantly improving convergence and performance.

Contribution

The paper proposes a novel Anderson-type acceleration technique specifically designed for stochastic gradient descent in deep neural network optimization.

Findings

Improved convergence speed in DNN training.

Effective acceleration for CNN optimization.

Demonstrated applicability across different neural network architectures.

Abstract

In this paper we consider the neural network optimization. We develop Anderson-type acceleration method for the stochastic gradient decent method and it improves the network permanence very much. We demonstrate the applicability of the method for Deep Neural Network (DNN) and Convolution Neural Network (CNN).