An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints

TL;DR

This paper uses information theory to quantify the performance loss caused by decentralization in privacy-constrained federated learning, considering both learning and inference phases across multiple agents.

Contribution

It introduces a novel information-theoretic framework to measure the cost of decentralization in federated learning with privacy constraints, applicable to general supervised problems.

Findings

Cost of decentralization is quantified via conditional mutual information.

Framework applies to any number of agents and supervised learning problems.

Provides theoretical bounds on performance loss due to decentralization.

Abstract

In vertical federated learning (FL), the features of a data sample are distributed across multiple agents. As such, inter-agent collaboration can be beneficial not only during the learning phase, as is the case for standard horizontal FL, but also during the inference phase. A fundamental theoretical question in this setting is how to quantify the cost, or performance loss, of decentralization for learning and/or inference. In this paper, we consider general supervised learning problems with any number of agents, and provide a novel information-theoretic quantification of the cost of decentralization in the presence of privacy constraints on inter-agent communication within a Bayesian framework. The cost of decentralization for learning and/or inference is shown to be quantified in terms of conditional mutual information terms involving features and label variables.