A Softmax-free Loss Function Based on Predefined Optimal-distribution of Latent Features for Deep Learning Classifier

TL;DR

This paper introduces POD Loss, a novel Softmax-free loss function based on predefined optimal-distribution of latent features, improving classification accuracy and convergence in deep learning classifiers.

Contribution

It proposes a new loss function that relies solely on latent features and predefined class centroids, eliminating the need for Softmax and enhancing performance.

Findings

POD Loss outperforms traditional Softmax and related loss functions in accuracy.

Models using POD Loss converge faster during training.

Experimental results on multiple datasets validate the effectiveness of POD Loss.

Abstract

In the field of pattern classification, the training of deep learning classifiers is mostly end-to-end learning, and the loss function is the constraint on the final output (posterior probability) of the network, so the existence of Softmax is essential. In the case of end-to-end learning, there is usually no effective loss function that completely relies on the features of the middle layer to restrict learning, resulting in the distribution of sample latent features is not optimal, so there is still room for improvement in classification accuracy. Based on the concept of Predefined Evenly-Distributed Class Centroids (PEDCC), this article proposes a Softmax-free loss function based on predefined optimal-distribution of latent features-POD Loss. The loss function only restricts the latent features of the samples, including the norm-adaptive Cosine distance between the latent feature vector of the sample and the center of the predefined evenly-distributed class, and the correlation between the latent features of the samples. Finally, Cosine distance is used for classification. Compared with the commonly used Softmax Loss, some typical Softmax related loss functions and PEDCC-Loss, experiments on several commonly used datasets on several typical deep learning classification networks show that the classification performance of POD Loss is always significant better and easier to converge. Code is available in https://github.com/TianYuZu/POD-Loss.