Revisiting lp-constrained Softmax Loss: A Comprehensive Study

TL;DR

This study comprehensively evaluates lp-constrained softmax loss classifiers, demonstrating their advantages in accuracy, overfitting resistance, and convergence across various datasets, architectures, and normalization parameters.

Contribution

It provides the first broad analysis of lp normalization effects on softmax classifiers across multiple tasks and architectures, highlighting its benefits for image classification.

Findings

Lp-constrained softmax classifiers improve accuracy.

They are less prone to overfitting.

Normalization enhances convergence and performance.

Abstract

Normalization is a vital process for any machine learning task as it controls the properties of data and affects model performance at large. The impact of particular forms of normalization, however, has so far been investigated in limited domain-specific classification tasks and not in a general fashion. Motivated by the lack of such a comprehensive study, in this paper we investigate the performance of lp-constrained softmax loss classifiers across different norm orders, magnitudes, and data dimensions in both proof-of-concept classification problems and real-world popular image classification tasks. Experimental results suggest collectively that lp-constrained softmax loss classifiers not only can achieve more accurate classification results but, at the same time, appear to be less prone to overfitting. The core findings hold across the three popular deep learning architectures tested and eight datasets examined, and suggest that lp normalization is a recommended data representation practice for image classification in terms of performance and convergence, and against overfitting.