Improving Quaternion Neural Networks with Quaternionic Activation Functions

TL;DR

This paper introduces novel quaternion activation functions that modify either magnitude or phase, leveraging quaternion properties to improve neural network performance, especially in image classification tasks, by enhancing gradient flow and utilizing quaternion algebra.

Contribution

The paper proposes new quaternion activation functions based on quaternion properties, offering an alternative to split activations and demonstrating improved performance in image classification.

Findings

Quaternion activation functions improve performance on CIFAR-10 and SVHN datasets.

Phase-modifying quaternion activations provide consistent performance gains.

Proposed functions enhance gradient flow in quaternion neural networks.

Abstract

In this paper, we propose novel quaternion activation functions where we modify either the quaternion magnitude or the phase, as an alternative to the commonly used split activation functions. We define criteria that are relevant for quaternion activation functions, and subsequently we propose our novel activation functions based on this analysis. Instead of applying a known activation function like the ReLU or Tanh on the quaternion elements separately, these activation functions consider the quaternion properties and respect the quaternion space $\mathbb{H}$. In particular, all quaternion components are utilized to calculate all output components, carrying out the benefit of the Hamilton product in e.g. the quaternion convolution to the activation functions. The proposed activation functions can be incorporated in arbitrary quaternion valued neural networks trained with gradient descent techniques. We further discuss the derivatives of the proposed activation functions where we observe beneficial properties for the activation functions affecting the phase. Specifically, they prove to be sensitive on basically the whole input range, thus improved gradient flow can be expected. We provide an elaborate experimental evaluation of our proposed quaternion activation functions including comparison with the split ReLU and split Tanh on two image classification tasks using the CIFAR-10 and SVHN dataset. There, especially the quaternion activation functions affecting the phase consistently prove to provide better performance.