An Investigation on Deep Learning with Beta Stabilizer

TL;DR

This paper extends the beta stabilizer method to LSTM and ReLU-based DNNs, demonstrating its ability to reduce learning rate sensitivity across different neural network architectures.

Contribution

It introduces the application of beta stabilizer to LSTM and ReLU-based DNNs, analyzing its impact on training stability and sensitivity to initial learning rate.

Findings

Beta stabilizer reduces learning rate sensitivity in DNN with ReLU.

It also decreases sensitivity in LSTM models.

Effects are less pronounced than in sigmoid-based DNNs.

Abstract

Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.