The Lack of Convexity of the Relevance-Compression Function

TL;DR

This paper explores the non-convex nature of the relevance-compression function in information bottleneck problems, linking it to phase transitions and challenging assumptions of convexity similar to rate-distortion curves.

Contribution

It demonstrates that the relevance-compression function can be non-convex and varies in convexity, unlike the convex rate-distortion curve, and relates this to phase transitions in the Lagrangian.

Findings

Relevance-compression function can be non-convex.

Convexity changes are linked to phase transitions.

The behavior differs from traditional rate-distortion curves.

Abstract

In this paper we investigate the convexity of the relevance-compression function for the Information Bottleneck and the Information Distortion problems. This curve is an analog of the rate-distortion curve, which is convex. In the problems we discuss in this paper, the distortion function is not a linear function of the quantizer, and the relevance-compression function is not necessarily convex (concave), but can change its convexity. We relate this phenomena with existence of first order phase transitions in the corresponding Lagrangian as a function of the annealing parameter.