A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

TL;DR

This paper introduces linearly convergent Douglas-Rachford splitting algorithms for Markovian information-theoretic problems like IB and PF, providing faster, more robust solutions with theoretical guarantees and empirical improvements over existing methods.

Contribution

The paper develops and proves the linear convergence of new Douglas-Rachford splitting algorithms for IB and PF, extending to convex-weakly convex objectives and surpassing existing solvers in performance.

Findings

IB solvers match benchmark solutions and converge over wider parameter ranges.

PF solvers better characterize privacy-utility trade-offs than greedy methods.

Proposed algorithms demonstrate faster convergence and broader applicability.

Abstract

In this work, we propose solving the Information bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that includes IB and PF into a unified framework. We prove the linear convergence of the proposed solvers using the Kurdyka-{\L}ojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of linearly convergent IB solvers, with one improves the performance of convergence over existing solvers while the other can be independent to the relevance-compression trade-off. Moreover, our results apply to PF, yielding a new class of linearly convergent PF solvers. Empirically, the proposed IB solvers IB obtain solutions that are comparable to the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficient than existing solvers. For PF, our non-greedy solvers can characterize the privacy-utility trade-off better than the clustering-based greedy solvers.