Empirical Strategy for Stretching Probability Distribution in Neural-network-based Regression

TL;DR

This paper introduces a novel loss function called weighted empirical stretching (WES) for neural network regression, improving distribution overlap and prediction accuracy across various distribution shapes and noise levels.

Contribution

The study proposes WES, a new loss function that enhances distribution overlap in neural network regression, applicable to any distribution shape and robust to noise.

Findings

WES outperforms traditional loss functions in distribution overlap.

WES improves RMSE in distribution tails, aiding abnormal event prediction.

Performance remains robust across different noise levels.

Abstract

In regression analysis under artificial neural networks, the prediction performance depends on determining the appropriate weights between layers. As randomly initialized weights are updated during back-propagation using the gradient descent procedure under a given loss function, the loss function structure can affect the performance significantly. In this study, we considered the distribution error, i.e., the inconsistency of two distributions (those of the predicted values and label), as the prediction error, and proposed weighted empirical stretching (WES) as a novel loss function to increase the overlap area of the two distributions. The function depends on the distribution of a given label, thus, it is applicable to any distribution shape. Moreover, it contains a scaling hyperparameter such that the appropriate parameter value maximizes the common section of the two distributions. To test the function capability, we generated ideal distributed curves (unimodal, skewed unimodal, bimodal, and skewed bimodal) as the labels, and used the Fourier-extracted input data from the curves under a feedforward neural network. In general, WES outperformed loss functions in wide use, and the performance was robust to the various noise levels. The improved results in RMSE for the extreme domain (i.e., both tail regions of the distribution) are expected to be utilized for prediction of abnormal events in non-linear complex systems such as natural disaster and financial crisis.