TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments

TL;DR

TinyProto introduces a novel prototype sparsification method for federated learning, significantly reducing communication costs while maintaining performance, especially suited for resource-limited and heterogeneous environments.

Contribution

It proposes Class-wise Prototype Sparsification and adaptive scaling to enhance communication efficiency in prototype-based federated learning.

Findings

Reduces communication costs by up to 4x compared to existing methods.

Maintains model performance despite significant compression.

Supports heterogeneous architectures without client-side computational overhead.

Abstract

Communication efficiency in federated learning (FL) remains a critical challenge for resource-constrained environments. While prototype-based FL reduces communication overhead by sharing class prototypes-mean activations in the penultimate layer-instead of model parameters, its efficiency decreases with larger feature dimensions and class counts. We propose TinyProto, which addresses these limitations through Class-wise Prototype Sparsification (CPS) and adaptive prototype scaling. CPS enables structured sparsity by allocating specific dimensions to class prototypes and transmitting only non-zero elements, while adaptive scaling adjusts prototypes based on class distributions. Our experiments show TinyProto reduces communication costs by up to 4x compared to existing methods while maintaining performance. Beyond its communication efficiency, TinyProto offers crucial advantages: achieving compression without client-side computational overhead and supporting heterogeneous architectures, making it ideal for resource-constrained heterogeneous FL.