Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent

TL;DR

This paper introduces a hybrid coordinate descent algorithm combining line search and gradient methods, improving neural network training efficiency through adaptive parameter updates and parallelization.

Contribution

The paper proposes a novel hybrid coordinate descent algorithm that adaptively switches between line search and gradient updates based on gradient magnitude, enhancing efficiency and parallelizability.

Findings

Larger threshold values improve efficiency.

Parallelizable line search reduces computational time.

Hyperparameters significantly affect convergence.

Abstract

This paper presents a novel coordinate descent algorithm leveraging a combination of one-directional line search and gradient information for parameter updates for a squared error loss function. Each parameter undergoes updates determined by either the line search or gradient method, contingent upon whether the modulus of the gradient of the loss with respect to that parameter surpasses a predefined threshold. Notably, a larger threshold value enhances algorithmic efficiency. Despite the potentially slower nature of the line search method relative to gradient descent, its parallelizability facilitates computational time reduction. Experimental validation conducted on a 2-layer Rectified Linear Unit network with synthetic data elucidates the impact of hyperparameters on convergence rates and computational efficiency.