Deformed Statistics Formulation of the Information Bottleneck Method

TL;DR

This paper extends the information bottleneck method using Tsallis nonadditive statistics, introducing a generalized variational principle, a nonadditive distortion measure, and deriving new update equations, broadening its theoretical foundation.

Contribution

It formulates a nonadditive variational principle for the IB method within Tsallis statistics, deriving a q-deformed divergence and update equations without prior assumptions.

Findings

The generalized IB Lagrangian yields a q-deformed Kullback-Leibler divergence.

The q*-deformed free energy is proven to be non-negative and convex.

Derived update equations extend IB to Tsallis nonadditive statistics.

Abstract

The theoretical basis for a candidate variational principle for the information bottleneck (IB) method is formulated within the ambit of the generalized nonadditive statistics of Tsallis. Given a nonadditivity parameter $ q $, the role of the \textit{additive duality} of nonadditive statistics ($ q^*=2-q $) in relating Tsallis entropies for ranges of the nonadditivity parameter $ q < 1 $ and $ q > 1 $ is described. Defining $ X $, $ \tilde X $, and $ Y $ to be the source alphabet, the compressed reproduction alphabet, and, the \textit{relevance variable} respectively, it is demonstrated that minimization of a generalized IB (gIB) Lagrangian defined in terms of the nonadditivity parameter $ q^* $ self-consistently yields the \textit{nonadditive effective distortion measure} to be the \textit{$ q $-deformed} generalized Kullback-Leibler divergence: $ D_{K-L}^{q}[p(Y|X)||p(Y|\tilde X)] $. This result is achieved without enforcing any \textit{a-priori} assumptions. Next, it is proven that the $q^*-deformed $ nonadditive free energy of the system is non-negative and convex. Finally, the update equations for the gIB method are derived. These results generalize critical features of the IB method to the case of Tsallis statistics.