A Conjugate Gradient Method for Nonlinear Programming Problems using Caputo Fractional Gradients

TL;DR

This paper introduces a novel Caputo fractional conjugate gradient method for unconstrained optimization that leverages fractional derivatives to improve convergence speed and stability, demonstrated through numerical experiments including neural networks.

Contribution

It develops a new fractional gradient-based conjugate gradient method using Caputo derivatives, with convergence analysis and application to neural networks.

Findings

Achieves faster convergence than traditional methods.

Demonstrates stable performance in neural network training.

Converges at least linearly under mild assumptions.

Abstract

The article proposes a Caputo fractional conjugate gradient (CFCG) method for unconstrained optimization problems which is applicable to smooth as well as non-smooth problmes. The proposed method uses a non-adaptive version of the Caputo fractional derivative that provides integer-order derivatives information. A descent direction is obtained using the Caputo fractional gradients of two consecutive iterative points with a parameter ($\beta$). An inexact line search technique based on Armijo-Wolfe line conditions is used to find a suitable step length. Finally, a descent sequence is generated. The convergence results are derived under mild assumptions that ensuring of convergence is at least linear. Moreover, the convergence of the proposed method for quadratic functions is established through a Tikhonov-regularized formulation that can be interpreted as an extension of the least-squares approach. Finally, some numerical experiments, including neural network applications, are performed to justify that the proposed method achieves faster and more stable performance.