A Statistical Analysis of Deep Federated Learning for Intrinsically Low-dimensional Data

TL;DR

This paper investigates the generalization error of deep federated learning on low-dimensional data, revealing that convergence rates depend on intrinsic rather than ambient data dimensions, especially considering client heterogeneity.

Contribution

It provides a theoretical analysis of deep federated regression, highlighting the role of entropic dimension in convergence rates under heterogeneous client data.

Findings

Error rates depend on intrinsic data dimension, not ambient dimension.

Heterogeneity between clients affects convergence, characterized by parameter Δ.

Error bounds differ for participating and non-participating clients.

Abstract

Despite significant research on the optimization aspects of federated learning, the exploration of generalization error, especially in the realm of heterogeneous federated learning, remains an area that has been insufficiently investigated, primarily limited to developments in the parametric regime. This paper delves into the generalization properties of deep federated regression within a two-stage sampling model. Our findings reveal that the intrinsic dimension, characterized by the entropic dimension, plays a pivotal role in determining the convergence rates for deep learners when appropriately chosen network sizes are employed. Specifically, when the true relationship between the response and explanatory variables is described by a $\beta$-H\"older function and one has access to $n$ independent and identically distributed (i.i.d.) samples from $m$ participating clients, for participating clients, the error rate scales at most as $\Tilde{O}((mn)^{-2\beta/(2\beta + \bar{d}_{2\beta}(\lambda))})$, whereas for non-participating clients, it scales as $\Tilde{O}(\Delta \cdot m^{-2\beta/(2\beta + \bar{d}_{2\beta}(\lambda))} + (mn)^{-2\beta/(2\beta + \bar{d}_{2\beta}(\lambda))})$. Here $\bar{d}_{2\beta}(\lambda)$ denotes the corresponding $2\beta$-entropic dimension of $\lambda$, the marginal distribution of the explanatory variables. The dependence between the two stages of the sampling scheme is characterized by $\Delta$. Consequently, our findings not only explicitly incorporate the ``heterogeneity" of the clients, but also highlight that the convergence rates of errors of deep federated learners are not contingent on the nominal high dimensionality of the data but rather on its intrinsic dimension.