Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks

TL;DR

This paper introduces a novel fractional-order Jacobian matrix differentiation method compatible with automatic differentiation, enhancing deep learning optimization by integrating fractional calculus into neural network training.

Contribution

It proposes a fractional-order Jacobian matrix differentiation method and fractional-order Autograd technology, enabling fractional differentiation in neural network layers for improved optimization.

Findings

Demonstrates superior performance of the proposed method in deep learning tasks.

Shows efficiency in training time and GPU memory usage.

Validates the effectiveness through experiments on multilayer perceptrons.

Abstract

Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due to insufficient theoretical research, at present, there is no fractional-order matrix differentiation method that is perfectly compatible with automatic differentiation (Autograd) technology. Therefore, we propose a fractional-order matrix differentiation calculation method. This method is introduced by the definition of the integer-order Jacobian matrix. We denote it as fractional-order Jacobian matrix differentiation (${{\bf{J}}^\alpha }$). Through ${{\bf{J}}^\alpha }$, we can carry out the matrix-based fractional-order chain rule. Based on the Linear module and the fractional-order differentiation, we design the fractional-order Autograd technology to enable the use of fractional-order differentiation in hidden layers, thereby enhancing the practicality of fractional-order differentiation in deep learning. In the experiment, according to the PyTorch framework, we design fractional-order Linear (FLinear) and replace nn.Linear in the multilayer perceptron with FLinear. Through the qualitative analysis of the training set and validation set $Loss$, the quantitative analysis of the test set indicators, and the analysis of time consumption and GPU memory usage during model training, we verify the superior performance of ${{\bf{J}}^\alpha }$ and prove that it is an excellent fractional-order gradient descent method in the field of deep learning.