Introducing the DOME Activation Functions

TL;DR

This paper introduces DOME, a new non-linear activation function that enhances class-compactness, regularization, and robustness in neural networks, serving as an alternative to traditional functions like sigmoid and softmax.

Contribution

The paper presents DOME, a novel activation function that can replace sigmoid, tanh, and softmax, and can be extended for intermediate layers, improving robustness and embedding properties.

Findings

DOME induces class-compactness and regularization.

Models with DOME show increased robustness against adversarial attacks.

DOME can be extended to various network layers and classification tasks.

Abstract

In this paper, we introduce a novel non-linear activation function that spontaneously induces class-compactness and regularization in the embedding space of neural networks. The function is dubbed DOME for Difference Of Mirrored Exponential terms. The basic form of the function can replace the sigmoid or the hyperbolic tangent functions as an output activation function for binary classification problems. The function can also be extended to the case of multi-class classification, and used as an alternative to the standard softmax function. It can also be further generalized to take more flexible shapes suitable for intermediate layers of a network. We empirically demonstrate the properties of the function. We also show that models using the function exhibit extra robustness against adversarial attacks.