Hybridised Loss Functions for Improved Neural Network Generalisation

TL;DR

This paper investigates hybrid loss functions combining cross entropy and sum squared error to enhance neural network generalisation, demonstrating improved performance across various problems.

Contribution

It introduces and evaluates hybrid loss functions that switch from sum squared error to cross entropy during training, showing improved generalisation in neural networks.

Findings

Hybrid loss functions improve neural network generalisation.

Switching from sum squared error to cross entropy yields best results.

Hybrid approach outperforms individual loss functions in experiments.

Abstract

Loss functions play an important role in the training of artificial neural networks (ANNs), and can affect the generalisation ability of the ANN model, among other properties. Specifically, it has been shown that the cross entropy and sum squared error loss functions result in different training dynamics, and exhibit different properties that are complementary to one another. It has previously been suggested that a hybrid of the entropy and sum squared error loss functions could combine the advantages of the two functions, while limiting their disadvantages. The effectiveness of such hybrid loss functions is investigated in this study. It is shown that hybridisation of the two loss functions improves the generalisation ability of the ANNs on all problems considered. The hybrid loss function that starts training with the sum squared error loss function and later switches to the cross entropy error loss function is shown to either perform the best on average, or to not be significantly different than the best loss function tested for all problems considered. This study shows that the minima discovered by the sum squared error loss function can be further exploited by switching to cross entropy error loss function. It can thus be concluded that hybridisation of the two loss functions could lead to better performance in ANNs.