# Heterogeneity-Aware Personalized Federated Neural Architecture Search

## TL;DR

This paper introduces a method to design personalized machine learning models in federated learning settings where devices and data vary widely.

## Contribution

The novel HAPFNAS method addresses both resource and statistical heterogeneity in federated NAS using knowledge distillation and performance predictors.

## Key findings

- HAPFNAS improves training stability and evaluation accuracy in heterogeneous federated environments.
- The proposed method outperforms existing federated NAS approaches on standard image classification datasets.
- HeteroFedAvg enhances collaborative training for personalized models in non-IID data settings.

## Abstract

Federated learning (FL), which enables collaborative learning across distributed nodes, confronts a significant heterogeneity challenge, primarily including resource heterogeneity induced by different hardware platforms, and statistical heterogeneity originating from non-IID private data distributions among clients. Neural architecture search (NAS), particularly one-shot NAS, holds great promise for automatically designing optimal personalized models tailored to such heterogeneous scenarios. However, the coexistence of both resource and statistical heterogeneity destabilizes the training of the one-shot supernet, impairs the evaluation of candidate architectures, and ultimately hinders the discovery of optimal personalized models. To address this problem, we propose a heterogeneity-aware personalized federated NAS (HAPFNAS) method. First, we leverage lightweight knowledge models to distill knowledge from clients to server-side supernet, thereby effectively mitigating the effects of heterogeneity and enhancing the training stability. Then, we build random-forest-based personalized performance predictors to enable the efficient evaluation of candidate architectures across clients. Furthermore, we develop a model-heterogeneous FL algorithm called heteroFedAvg to facilitate collaborative model training for the discovered personalized models. Comprehensive experiments on CIFAR-10/100 and Tiny-ImageNet classification datasets demonstrate the effectiveness of our HAPFNAS, compared to state-of-the-art federated NAS methods.