Rethinking Feature Distribution for Loss Functions in Image Classification

TL;DR

This paper introduces a large-margin Gaussian Mixture loss for deep neural networks that improves classification accuracy and enables detection of abnormal inputs by modeling feature distributions more effectively.

Contribution

The paper proposes a novel L-GM loss based on Gaussian Mixture assumptions, enhancing classification performance and abnormal input detection over traditional softmax loss.

Findings

Outperforms softmax loss on multiple benchmarks

Effectively detects adversarial examples

Models feature distribution accurately

Abstract

We propose a large-margin Gaussian Mixture (L-GM) loss for deep neural networks in classification tasks. Different from the softmax cross-entropy loss, our proposal is established on the assumption that the deep features of the training set follow a Gaussian Mixture distribution. By involving a classification margin and a likelihood regularization, the L-GM loss facilitates both a high classification performance and an accurate modeling of the training feature distribution. As such, the L-GM loss is superior to the softmax loss and its major variants in the sense that besides classification, it can be readily used to distinguish abnormal inputs, such as the adversarial examples, based on their features' likelihood to the training feature distribution. Extensive experiments on various recognition benchmarks like MNIST, CIFAR, ImageNet and LFW, as well as on adversarial examples demonstrate the effectiveness of our proposal.