Unbounded Output Networks for Classification

TL;DR

This paper introduces UnBounded output networks (UBnets), a novel neural network architecture with unbounded outputs and a modified training objective, achieving superior classification performance and robustness against adversarial attacks on benchmark datasets.

Contribution

The paper proposes UBnets, integrating unbounded outputs and a new training method, improving classification accuracy and adversarial robustness over standard neural networks.

Findings

UBnets outperform standard neural networks on MNIST, CIFAR-10, and CIFAR-100.

Setting target value to the number of hidden units enhances performance.

UBnets demonstrate increased robustness against adversarial examples.

Abstract

We proposed the expected energy-based restricted Boltzmann machine (EE-RBM) as a discriminative RBM method for classification. Two characteristics of the EE-RBM are that the output is unbounded and that the target value of correct classification is set to a value much greater than one. In this study, by adopting features of the EE-RBM approach to feed-forward neural networks, we propose the UnBounded output network (UBnet) which is characterized by three features: (1) unbounded output units; (2) the target value of correct classification is set to a value much greater than one; and (3) the models are trained by a modified mean-squared error objective. We evaluate our approach using the MNIST, CIFAR-10, and CIFAR-100 benchmark datasets. We first demonstrate, for shallow UBnets on MNIST, that a setting of the target value equal to the number of hidden units significantly outperforms a setting of the target value equal to one, and it also outperforms standard neural networks by about 25\%. We then validate our approach by achieving high-level classification performance on the three datasets using unbounded output residual networks. We finally use MNIST to analyze the learned features and weights, and we demonstrate that UBnets are much more robust against adversarial examples than the standard approach of using a softmax output layer and training the networks by a cross-entropy objective.