General transformation neural networks: A class of parametrized functions for high-dimensional function approximation

TL;DR

This paper introduces general transformation neural networks (GTNNs), a new class of neural networks with enhanced capacity for high-dimensional function approximation, especially for oscillatory functions, outperforming traditional neural networks in accuracy.

Contribution

The paper proposes GTNNs with generalized neuron transformations, providing theoretical analysis of their approximation capabilities and demonstrating superior empirical performance.

Findings

GTNNs have universal approximation properties.

GTNNs achieve lower approximation errors on oscillatory functions.

Numerical examples show GTNNs outperform conventional neural networks.

Abstract

We propose a novel class of neural network-like parametrized functions, i.e., general transformation neural networks (GTNNs), for high-dimensional approximation. Conventional deep neural networks sometimes perform less accurately on learning problems trained with gradient descent, especially when the target function is oscillatory. To improve accuracy, we generalize the neuron's affine transformation to a broader class of functions that can capture complex shapes and offer greater capacity. Specifically, we discuss three types of GTNNs in detail: the cubic, quadratic and trigonometric transformation neural networks (CTNNs, QTNNs and TTNNs). We perform an approximation error analysis of GTNNs, presenting their universal approximation properties for continuous functions, error bounds for Barron-type functions and error bounds of deep architectures. Several numerical examples of regression problems are presented, demonstrating that CTNNs/QTNNs/TTNNs achieve higher accuracy than conventional fully connected neural networks.