Brownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network

TL;DR

This paper introduces BrownianReLU, a stochastic activation function based on Brownian motion, designed to improve gradient stability and learning in LSTM networks for noisy financial data.

Contribution

The paper proposes BrownianReLU, a novel stochastic activation function that enhances gradient propagation and stability in LSTMs, especially for non-stationary financial time series.

Findings

BrownianReLU improves predictive accuracy on financial data.

It reduces the dying ReLU problem in LSTMs.

Activation choice impacts classification sensitivity and accuracy.

Abstract

Deep learning models are effective for sequential data modeling, yet commonly used activation functions such as ReLU, LeakyReLU, and PReLU often exhibit gradient instability when applied to noisy, non-stationary financial time series. This study introduces BrownianReLU, a stochastic activation function induced by Brownian motion that enhances gradient propagation and learning stability in Long Short-Term Memory (LSTM) networks. Using Monte Carlo simulation, BrownianReLU provides a smooth, adaptive response for negative inputs, mitigating the dying ReLU problem. The proposed activation is evaluated on financial time series from Apple, GCB, and the S&P 500, as well as LendingClub loan data for classification. Results show consistently lower Mean Squared Error and higher $R^2$ values, indicating improved predictive accuracy and generalization. Although ROC-AUC metric is limited in classification tasks, activation choice significantly affects the trade-off between accuracy and sensitivity, with Brownian ReLU and the selected activation functions yielding practically meaningful performance.