Enhancing Cross Entropy with a Linearly Adaptive Loss Function for Optimized Classification Performance

TL;DR

This paper introduces a novel linearly adaptive cross entropy loss function that improves classification accuracy over standard cross entropy while maintaining computational efficiency, demonstrated on CIFAR-100 with ResNet.

Contribution

The paper presents a new adaptive loss function derived from information theory that enhances optimization in classification tasks with minimal added complexity.

Findings

Outperforms standard cross entropy in accuracy on CIFAR-100

Maintains similar computational efficiency as traditional cross entropy

Potential to inspire future loss function research

Abstract

We propose the Linearly Adaptive Cross Entropy Loss function. This is a novel measure derived from the information theory. In comparison to the standard cross entropy loss function, the proposed one has an additional term that depends on the predicted probability of the true class. This feature serves to enhance the optimization process in classification tasks involving one-hot encoded class labels. The proposed one has been evaluated on a ResNet-based model using the CIFAR-100 dataset. Preliminary results show that the proposed one consistently outperforms the standard cross entropy loss function in terms of classification accuracy. Moreover, the proposed one maintains simplicity, achieving practically the same efficiency to the traditional cross entropy loss. These findings suggest that our approach could broaden the scope for future research into loss function design.