Making Sigmoid-MSE Great Again: Output Reset Challenges Softmax Cross-Entropy in Neural Network Classification

TL;DR

This paper compares MSE and Softmax Cross-Entropy for neural network classification, introducing an Output Reset algorithm that improves robustness and shows MSE can be a viable alternative to SCE, especially with noisy data.

Contribution

The study introduces the Output Reset algorithm and demonstrates MSE with sigmoid activation as a competitive alternative to SCE in classification tasks.

Findings

MSE with sigmoid achieves comparable accuracy to SCE.

Output Reset reduces errors and improves robustness.

MSE performs better with noisy data.

Abstract

This study presents a comparative analysis of two objective functions, Mean Squared Error (MSE) and Softmax Cross-Entropy (SCE) for neural network classification tasks. While SCE combined with softmax activation is the conventional choice for transforming network outputs into class probabilities, we explore an alternative approach using MSE with sigmoid activation. We introduce the Output Reset algorithm, which reduces inconsistent errors and enhances classifier robustness. Through extensive experiments on benchmark datasets (MNIST, CIFAR-10, and Fashion-MNIST), we demonstrate that MSE with sigmoid activation achieves comparable accuracy and convergence rates to SCE, while exhibiting superior performance in scenarios with noisy data. Our findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.