Federated Representation Learning via Maximal Coding Rate Reduction

TL;DR

This paper introduces FLOW, a federated learning method that leverages maximal coding rate reduction to learn low-dimensional, discriminative data representations across multiple clients, moving beyond traditional cross-entropy loss.

Contribution

The paper presents a novel federated learning approach using MCR2 as the objective, with theoretical convergence guarantees and demonstrated practical utility.

Findings

Achieves a first-order stationary point in the distributed algorithm.

Produces representations that are both discriminative and compressible.

Numerical experiments validate the effectiveness of the learned representations.

Abstract

We propose a federated methodology to learn low-dimensional representations from a dataset that is distributed among several clients. In particular, we move away from the commonly-used cross-entropy loss in federated learning, and seek to learn shared low-dimensional representations of the data in a decentralized manner via the principle of maximal coding rate reduction (MCR2). Our proposed method, which we refer to as FLOW, utilizes MCR2 as the objective of choice, hence resulting in representations that are both between-class discriminative and within-class compressible. We theoretically show that our distributed algorithm achieves a first-order stationary point. Moreover, we demonstrate, via numerical experiments, the utility of the learned low-dimensional representations.