Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems

TL;DR

This paper investigates the phenomenon of critical slowing down in the convergence of Arimoto-Blahut algorithms near topological transitions in rate-distortion problems, highlighting its implications for machine learning and data compression.

Contribution

It identifies and analyzes the critical slowing down phenomenon in the convergence of algorithms solving rate-distortion and information bottleneck problems near topological transitions.

Findings

Convergence time diverges near critical points.

Critical slowing down affects practical algorithm performance.

Implications for machine learning and data compression.

Abstract

In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the rate-distortion and information bottleneck settings. We argue that they suffer from critical slowing down -- a diverging number of iterations for convergence -- near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.