Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations

TL;DR

This paper introduces Separable Gaussian Neural Networks (SGNN), a computationally efficient alternative to traditional Gaussian neural networks that maintains accuracy and improves speed, especially in high-dimensional function approximation tasks.

Contribution

The paper proposes SGNN, a novel neural network structure leveraging Gaussian separability to significantly reduce computational complexity while preserving accuracy.

Findings

SGNN achieves 100x speedup over GRBFNN in 3D function approximation.

SGNN maintains similar accuracy to GRBFNN despite reduced complexity.

SGNN outperforms DNNs with ReLU and Sigmoid in complex function approximation by three orders of magnitude.

Abstract

The Gaussian-radial-basis function neural network (GRBFNN) has been a popular choice for interpolation and classification. However, it is computationally intensive when the dimension of the input vector is high. To address this issue, we propose a new feedforward network - Separable Gaussian Neural Network (SGNN) by taking advantage of the separable property of Gaussian functions, which splits input data into multiple columns and sequentially feeds them into parallel layers formed by uni-variate Gaussian functions. This structure reduces the number of neurons from O(N^d) of GRBFNN to O(dN), which exponentially improves the computational speed of SGNN and makes it scale linearly as the input dimension increases. In addition, SGNN can preserve the dominant subspace of the Hessian matrix of GRBFNN in gradient descent training, leading to a similar level of accuracy to GRBFNN. It is experimentally demonstrated that SGNN can achieve 100 times speedup with a similar level of accuracy over GRBFNN on tri-variate function approximations. The SGNN also has better trainability and is more tuning-friendly than DNNs with RuLU and Sigmoid functions. For approximating functions with complex geometry, SGNN can lead to three orders of magnitude more accurate results than a RuLU-DNN with twice the number of layers and the number of neurons per layer.