FEATHERS: Federated Architecture and Hyperparameter Search

TL;DR

FEATHERS is a federated approach for neural architecture and hyperparameter search that maintains data privacy using differential privacy, enabling efficient joint optimization in distributed settings without sacrificing model performance.

Contribution

It introduces a novel federated method for joint neural architecture and hyperparameter search that respects privacy constraints via differential privacy.

Findings

Efficiently optimizes architectures and hyperparameters in federated settings.

Converges on classification tasks without performance loss under privacy constraints.

Maintains data privacy while enabling effective neural architecture search.

Abstract

Deep neural architectures have profound impact on achieved performance in many of today's AI tasks, yet, their design still heavily relies on human prior knowledge and experience. Neural architecture search (NAS) together with hyperparameter optimization (HO) helps to reduce this dependence. However, state of the art NAS and HO rapidly become infeasible with increasing amount of data being stored in a distributed fashion, typically violating data privacy regulations such as GDPR and CCPA. As a remedy, we introduce FEATHERS - $\textbf{FE}$derated $\textbf{A}$rchi$\textbf{T}$ecture and $\textbf{H}$yp$\textbf{ER}$parameter $\textbf{S}$earch, a method that not only optimizes both neural architectures and optimization-related hyperparameters jointly in distributed data settings, but further adheres to data privacy through the use of differential privacy (DP). We show that FEATHERS efficiently optimizes architectural and optimization-related hyperparameters alike, while demonstrating convergence on classification tasks at no detriment to model performance when complying with privacy constraints.