Enhancing Classifier Conservativeness and Robustness by Polynomiality

TL;DR

This paper introduces polynomiality as a means to improve classifier conservativeness and robustness, proposing softRmax as a polynomial-based alternative to softmax, enhancing adversarial robustness and reducing overconfidence.

Contribution

The paper presents a novel polynomial approach to classifier decision functions and introduces softRmax, a new softmax variant derived from Gaussian models, improving robustness.

Findings

Polynomiality reduces overconfidence in classifiers.

softRmax enhances robustness against adversarial attacks.

The approach maintains performance in the tails of data distribution.

Abstract

We illustrate the detrimental effect, such as overconfident decisions, that exponential behavior can have in methods like classical LDA and logistic regression. We then show how polynomiality can remedy the situation. This, among others, leads purposefully to random-level performance in the tails, away from the bulk of the training data. A directly related, simple, yet important technical novelty we subsequently present is softRmax: a reasoned alternative to the standard softmax function employed in contemporary (deep) neural networks. It is derived through linking the standard softmax to Gaussian class-conditional models, as employed in LDA, and replacing those by a polynomial alternative. We show that two aspects of softRmax, conservativeness and inherent gradient regularization, lead to robustness against adversarial attacks without gradient obfuscation.