CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
Feihu Huang, Guanyi Zhang, Songcan Chen

TL;DR
This paper introduces CLion, an improved Cautious Lion optimizer with better generalization and convergence properties, supported by theoretical analysis and extensive experiments.
Contribution
It proposes a novel Cautious Lion optimizer that enhances generalization and convergence, with rigorous theoretical proofs and empirical validation.
Findings
CLion achieves lower generalization error $O(1/N)$ compared to Lion.
The generalization error of Lion is $O(1/(N au^T))$, which CLion improves upon.
Extensive experiments confirm the effectiveness of CLion in practice.
Abstract
Lion optimizer is a popular learning-based optimization algorithm in machine learning, which shows impressive performance in training many deep learning models. Although convergence property of the Lion optimizer has been studied, its generalization analysis is still missing. To fill this gap, we study generalization property of the Lion via algorithmic stability based on the mathematical induction. Specifically, we prove that the Lion has a generalization error of , where is training sample size, and denotes the smallest absolute value of non-zero element in gradient estimator, and is the total iteration number. In addition, we obtain an interesting byproduct that the SignSGD algorithm has the same generalization error as the Lion. To enhance generalization of the Lion, we design a novel efficient Cautious Lion (i.e., CLion) optimizer by…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
