FMCL: Class-Aware Client Clustering with Foundation Model Representations for Heterogeneous Federated Learning

TL;DR

FMCL introduces a one-shot, class-aware client clustering method using foundation model representations to improve federated learning performance under data heterogeneity.

Contribution

It proposes a novel one-shot clustering framework leveraging foundation models for class-aware client grouping in federated learning.

Findings

FMCL improves federated learning accuracy on heterogeneous benchmarks.

FMCL achieves more stable clustering behavior compared to existing methods.

Clustering is performed once without additional communication during training.

Abstract

Federated Learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its performance deteriorates under statistical heterogeneity. Clustered Federated Learning addresses this challenge by grouping similar clients and training separate models per cluster. However, existing clustering strategies often rely on raw data statistics, model parameters, or heuristic similarity measures that fail to capture class-level semantic structure across heterogeneous domains and frequently require iterative coordination. We propose FMCL, a one-shot, class-aware client clustering framework that leverages foundation model representations to construct semantic client signatures. Using a frozen foundation model, FMCL computes class-level embedding prototypes for each client and measures similarity via cosine distance between their class-aware representations. Clustering is performed once prior to training, introducing no additional communication during federated optimization and remaining agnostic to the downstream model architecture. Extensive experiments across heterogeneous benchmarks demonstrate that FMCL improves federated performance and yields more stable clustering behavior compared to existing clustering-based methods under non-identically distributed data partitioning.