Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimization

TL;DR

This paper introduces HiSo, a Hessian-informed zeroth-order federated optimization method that accelerates convergence and reduces communication costs in federated learning, especially for large language models, by leveraging global Hessian approximations without increasing communication overhead.

Contribution

The paper proposes HiSo, a novel Hessian-informed ZO federated optimization method that accelerates convergence using global Hessian approximations while maintaining scalar-only communication.

Findings

HiSo achieves 1-5x faster convergence in communication rounds compared to state-of-the-art ZO-FL methods.

Theoretically, HiSo's convergence rate is independent of the Lipschitz constant and model dimension under certain assumptions.

Empirical results demonstrate significant communication savings and faster fine-tuning of large language models.

Abstract

Zeroth-order (ZO) optimization enables dimension-free communication in federated learning (FL), making it attractive for fine-tuning of large language models (LLMs) due to significant communication savings. However, existing ZO-FL methods largely overlook curvature information, despite its well-established benefits for convergence acceleration. To address this, we propose HiSo, a Hessian-informed ZO federated optimization method that accelerates convergence by leveraging global diagonal Hessian approximations, while strictly preserving scalar-only communication without transmitting any second-order information. Theoretically, for non-convex functions, we show that HiSo can achieve an accelerated convergence rate that is independent of the Lipschitz constant $L$ and model dimension $d$ under some Hessian approximation assumptions, offering a plausible explanation for the observed phenomenon of ZO convergence being much faster than its worst-case $\mathscr{O}(d)$-bound. Empirically, across diverse LLM fine-tuning benchmarks, HiSo delivers a 1$\sim$5$\times$ speedup in communication rounds over existing state-of-the-art ZO-FL baselines. This superior convergence not only cuts communication costs but also provides strong empirical evidence that Hessian information acts as an effective accelerator in federated ZO optimization settings. Our source code is provided at https://github.com/ZidongLiu/DeComFL.