MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity

TL;DR

This paper introduces MP-GELU, a novel nonlinear function for Bayesian neural networks that allows faster and more efficient moment propagation, improving prediction accuracy and uncertainty quantification.

Contribution

The paper proposes MP-GELU, a new nonlinear function enabling analytical moment computation in BNNs, reducing computational costs compared to traditional functions like ReLU.

Findings

MP-GELU achieves faster moment propagation.

MP-GELU improves prediction accuracy.

MP-GELU enhances uncertainty estimation.

Abstract

Bayesian neural networks (BNNs) have been an important framework in the study of uncertainty quantification. Deterministic variational inference, one of the inference methods, utilizes moment propagation to compute the predictive distributions and objective functions. Unfortunately, deriving the moments requires computationally expensive Taylor expansion in nonlinear functions, such as a rectified linear unit (ReLU) or a sigmoid function. Therefore, a new nonlinear function that realizes faster moment propagation than conventional functions is required. In this paper, we propose a novel nonlinear function named moment propagating-Gaussian error linear unit (MP-GELU) that enables the fast derivation of first and second moments in BNNs. MP-GELU enables the analytical computation of moments by applying nonlinearity to the input statistics, thereby reducing the computationally expensive calculations required for nonlinear functions. In empirical experiments on regression tasks, we observed that the proposed MP-GELU provides higher prediction accuracy and better quality of uncertainty with faster execution than those of ReLU-based BNNs.