BrowNNe: Brownian Nonlocal Neurons & Activation Functions

TL;DR

This paper introduces Brownian Nonlocal Neurons, a novel stochastic activation function framework that leverages nonlocal derivatives and Brownian motion, demonstrating improved generalization in deep learning models, especially with limited training data.

Contribution

It develops a new theoretical framework for nonlocal derivatives, analyzes their properties, and applies Brownian motion-based stochastic activation functions to enhance deep learning performance.

Findings

Brownian motion-based activations outperform ReLU in low-data regimes.

Nonlocal derivatives are shown to be epsilon-sub gradients with convergence guarantees.

Theoretical analysis links nonlocal derivatives to Gaussian processes in Brownian motion.

Abstract

It is generally thought that the use of stochastic activation functions in deep learning architectures yield models with superior generalization abilities. However, a sufficiently rigorous statement and theoretical proof of this heuristic is lacking in the literature. In this paper, we provide several novel contributions to the literature in this regard. Defining a new notion of nonlocal directional derivative, we analyze its theoretical properties (existence and convergence). Second, using a probabilistic reformulation, we show that nonlocal derivatives are epsilon-sub gradients, and derive sample complexity results for convergence of stochastic gradient descent-like methods using nonlocal derivatives. Finally, using our analysis of the nonlocal gradient of Holder continuous functions, we observe that sample paths of Brownian motion admit nonlocal directional derivatives, and the nonlocal derivatives of Brownian motion are seen to be Gaussian processes with computable mean and standard deviation. Using the theory of nonlocal directional derivatives, we solve a highly nondifferentiable and nonconvex model problem of parameter estimation on image articulation manifolds. Using Brownian motion infused ReLU activation functions with the nonlocal gradient in place of the usual gradient during backpropagation, we also perform experiments on multiple well-studied deep learning architectures. Our experiments indicate the superior generalization capabilities of Brownian neural activation functions in low-training data regimes, where the use of stochastic neurons beats the deterministic ReLU counterpart.