Regularized Diffusion Adaptation via Conjugate Smoothing

TL;DR

This paper introduces a distributed diffusion strategy for Pareto optimization of regularized risks, using conjugate smoothing to handle non-differentiable regularizers and establishing broad applicability with weaker performance bounds.

Contribution

It proposes a novel distributed approach employing conjugate smoothing for non-differentiable regularizers in Pareto optimization, with theoretical performance guarantees.

Findings

Pareto solutions of smoothed problems approximate original solutions arbitrarily closely.

Performance bounds are established under weaker conditions than previous literature.

The method is applicable to a broader class of adaptation and learning problems.

Abstract

The purpose of this work is to develop and study a distributed strategy for Pareto optimization of an aggregate cost consisting of regularized risks. Each risk is modeled as the expectation of some loss function with unknown probability distribution while the regularizers are assumed deterministic, but are not required to be differentiable or even continuous. The individual, regularized, cost functions are distributed across a strongly-connected network of agents and the Pareto optimal solution is sought by appealing to a multi-agent diffusion strategy. To this end, the regularizers are smoothed by means of infimal convolution and it is shown that the Pareto solution of the approximate, smooth problem can be made arbitrarily close to the solution of the original, non-smooth problem. Performance bounds are established under conditions that are weaker than assumed before in the literature, and hence applicable to a broader class of adaptation and learning problems.