Smooth Mathematical Function from Compact Neural Networks

TL;DR

This paper introduces a new neural network approach for highly accurate and smooth function approximation, featuring a novel activation function, meta-batch method, and specialized data handling for regression tasks.

Contribution

It proposes a new activation function (ISLU), a meta-batch training method, and a neural network structure tailored for compact and smooth function approximation.

Findings

Neural networks can generate highly smooth functions with few parameters.

The meta-batch method significantly improves neural network performance.

The proposed approach effectively models mathematical functions with high accuracy.

Abstract

This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regression. In this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose a new activation function--integrated sigmoid linear unit (ISLU). Then special charateristics of metadata for regression, which is different from other data like image or sound, is discussed for improving the performance of neural networks. Finally, the one of a simple hierarchical NN that generate models substituting mathematical function is presented, and the new batch concept ``meta-batch" which improves the performance of NN several times more is introduced. The new activation function, meta-batch method, features of numerical data, meta-augmentation with metaparameters, and a structure of NN generating a compact multi-layer perceptron(MLP) are essential in this study.