HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation

TL;DR

HO-SFL introduces a hybrid federated learning framework that combines first-order server updates with zeroth-order client optimization, reducing memory and communication costs while maintaining fast convergence.

Contribution

This work presents a novel hybrid split federated learning method that decouples client and server optimization, enabling dimension-free aggregation and improved convergence.

Findings

HO-SFL reduces client memory usage by eliminating backpropagation.

HO-SFL achieves convergence speeds comparable to first-order methods.

HO-SFL significantly lowers communication costs in federated learning.

Abstract

Fine-tuning large models on edge devices is severely hindered by the memory-intensive backpropagation (BP) in standard frameworks like federated learning and split learning. While substituting BP with zeroth-order optimization can significantly reduce memory footprints, it typically suffers from prohibitively degraded convergence speed. To resolve this dilemma, we propose Hybrid-Order Split Federated Learning (HO-SFL). By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape: The server performs precise first-order updates (i.e., BP), whereas clients conduct memory-efficient zeroth-order optimization. This hybrid design not only eliminates the need for client-side BP but also enables dimension-free model aggregation, drastically lowering communication costs. Crucially, we provide a theoretical convergence analysis, demonstrating that HO-SFL mitigates the dimension-dependent convergence slowdown of zeroth-order optimization, achieving a convergence rate comparable to first-order methods. Extensive experiments on tasks across vision and language modalities validate that HO-SFL achieves convergence speeds comparable to first-order baselines while significantly reducing communication costs and client memory footprints.