An Algorithm for Computing the Capacity of Symmetrized KL Information for Discrete Channels

TL;DR

This paper introduces an algorithm to compute the capacity of discrete channels based on symmetrized KL information, addressing non-concavity challenges and validating results on various channel types.

Contribution

The paper presents a novel iterative algorithm for calculating symmetrized KL capacity, handling non-concavity and applying it to multiple channel models including machine learning scenarios.

Findings

Algorithm accurately computes capacity for Binary symmetric and Binomial channels.

Method demonstrates consistency with theoretical capacity values.

Effective in identifying worst-case distributions in machine learning applications.

Abstract

Symmetrized Kullback-Leibler (KL) information (\(I_{\mathrm{SKL}}\)), which symmetrizes the traditional mutual information by integrating Lautum information, has been shown as a critical quantity in communication~\cite{aminian2015capacity} and learning theory~\cite{aminian2023information}. This paper considers the problem of computing the capacity in terms of \(I_{\mathrm{SKL}}\) for a fixed discrete channel. Such a maximization problem is reformulated into a discrete quadratic optimization with a simplex constraint. One major challenge here is the non-concavity of Lautum information, which complicates the optimization problem. Our method involves symmetrizing the KL divergence matrix and applying iterative updates to ensure a non-decreasing update while maintaining a valid probability distribution. We validate our algorithm on Binary symmetric Channels and Binomial Channels, demonstrating its consistency with theoretical values. Additionally, we explore its application in machine learning through the Gibbs channel, showcasing the effectiveness of our algorithm in finding the worst-case data distributions.