Revisit the Arimoto-Blahut algorithm: New Analysis with Approximation

TL;DR

This paper revisits the Arimoto-Blahut algorithm, providing a new analysis that shows its convergence rate to the channel capacity is inverse exponential, improving understanding of its efficiency in approximating the capacity.

Contribution

The paper offers a novel convergence rate analysis of the Arimoto-Blahut algorithm, demonstrating inverse exponential convergence towards the capacity, especially for approximate solutions.

Findings

Convergence rate is inverse exponential for small approximation errors.

New bounds on the rate of convergence for capacity computation.

Enhanced understanding of the algorithm's efficiency in practical scenarios.

Abstract

By the seminal paper of Claude Shannon \cite{Shannon48}, the computation of the capacity of a discrete memoryless channel has been considered as one of the most important and fundamental problems in Information Theory. Nearly 50 years ago, Arimoto and Blahut independently proposed identical algorithms to solve this problem in their seminal papers \cite{Arimoto1972AnAF, Blahut1972ComputationOC}. The Arimoto-Blahut algorithm was proven to converge to the capacity of the channel as $t \to \infty$, with a convergence rate upper bounded by $O\left(\log(m)/t\right)$, where $m$ is the size of the input distribution. Under the assumption that a unique optimal solution is in the interior of the input probability simplex, the convergence becomes inverse exponential after an iteration $t^0$ \cite{Arimoto1972AnAF}. More recently, it was demonstrated in \cite{Nakagawa2020AnalysisOT} that in certain specific cases, the convergence rate is at worst case inverse linear. In this paper, we revisit this fundamental algorithm analyzing its rate of convergence focusing on the approximation of the capacity. Our main result shows that the convergence rate to an $\varepsilon$-optimal solution, for any sufficiently small constant $\varepsilon > 0$, is inverse exponential $O\left(\log(m)/c^t\right)$, for some constant $c > 1$. Given this, we derive new and complementary results for the computation of capacity, particularly in cases where an exact solution is sought.