OCP-GN: A Scalable Second-order Optimizer for Stochastic Optimization

TL;DR

This paper introduces OCP-GN, a scalable second-order optimizer based on the Optimal Control Principle, designed for large-scale neural network training with demonstrated superior performance.

Contribution

The paper presents a novel second-order optimization algorithm with linear complexity and robustness, outperforming existing methods in neural network training.

Findings

OCP-GN achieves better convergence than traditional optimizers.

The algorithm maintains linear computational complexity.

Experimental results show significant performance improvements.

Abstract

This paper proposes a novel second-order optimization algorithm based on the Optimal Control Principle (OCP), applicable to large-scale optimization problems in neural network training. The algorithm has a computational complexity of O(d) and strong robustness. Extensive experiments on multiple benchmarks demonstrate the significant superiority of the proposed method.