Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings

TL;DR

This paper provides a detailed convergence analysis of the Lion optimizer, introducing variance reduction and communication-efficient variants, with theoretical convergence rates for both centralized and distributed settings.

Contribution

It presents the first convergence rate analysis of the Lion optimizer, including variance reduction and communication-efficient variants in distributed environments.

Findings

Standard Lion optimizer has a convergence rate of O(d^{1/2} T^{-1/4})

Variance reduction improves the rate to O(d^{1/2} T^{-1/3})

Distributed Lion achieves rates of O(d^{1/2}(nT)^{-1/4}) and O(d^{1/2}(nT)^{-1/3})

Abstract

In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of $\mathcal{O}(d^{1/2}T^{-1/4})$ under standard assumptions, where $d$ denotes the problem dimension and $T$ is the iteration number. To further improve this rate, we introduce the Lion optimizer with variance reduction, resulting in an enhanced convergence rate of $\mathcal{O}(d^{1/2}T^{-1/3})$. We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of $\mathcal{O}(d^{1/2}(nT)^{-1/4})$ and $\mathcal{O}(d^{1/2}(nT)^{-1/3})$, with $n$ denoting the number of nodes. Furthermore, we investigate a communication-efficient variant of the distributed Lion that ensures sign compression in both communication directions. By employing the unbiased sign operations, the proposed Lion variant and its variance reduction counterpart, achieve convergence rates of $\mathcal{O}\left( \max \left\{\frac{d^{1/4}}{T^{1/4}}, \frac{d^{1/10}}{n^{1/5}T^{1/5}} \right\} \right)$ and $\mathcal{O}\left( \frac{d^{1/4}}{T^{1/4}} \right)$, respectively.