A Low-Rank Symplectic Gradient Adjustment Method for Computing Nash Equilibria

TL;DR

This paper introduces a low-rank symplectic gradient adjustment (LRSGA) method that efficiently computes Nash equilibria by approximating second-order derivatives, significantly reducing computational effort compared to traditional SGA.

Contribution

The paper develops a low-rank approximation of the SGA method with theoretical convergence analysis and demonstrates its superior efficiency in neural network training.

Findings

LRSGA outperforms SGA in computational efficiency

Theoretical convergence bounds are established for LRSGA

LRSGA achieves orders of magnitude faster training in neural networks

Abstract

This work presents a theoretical and numerical investigation of the symplectic gradient adjustment (SGA) method and of a low-rank SGA (LRSGA) method for efficiently solving two-objective optimization problems in the framework of Nash games. The SGA method outperforms the gradient method by including second-order mixed derivatives computed at each iterate, which requires considerably larger computational effort. For this reason, a LRSGA method is proposed where the approximation to second-order mixed derivatives are obtained by rank-one updates. The theoretical analysis presented in this work focuses on novel convergence estimates for the SGA and LRSGA methods, including parameter bounds. The superior computational complexity of the LRSGA method is demonstrated in the training of a CLIP neural architecture, where the LRSGA method outperforms the SGA method by orders of magnitude smaller CPU time.