On the Hardness of Entropy Minimization and Related Problems

TL;DR

This paper explores the computational complexity of entropy-related optimization problems, proving NP-hardness in various scenarios and introducing new metrics with similar complexity challenges.

Contribution

It provides simple proofs of NP-hardness for entropy minimization and mutual information maximization over transportation polytopes and introduces new entropy-based metrics with proven computational difficulty.

Findings

NP-hardness of entropy minimization and mutual information maximization over transportation polytopes

Complexity differences among entropy problems on specific polytopes

Introduction of new entropy-based metrics that are NP-hard to compute

Abstract

We investigate certain optimization problems for Shannon information measures, namely, minimization of joint and conditional entropies $H(X,Y)$, $H(X|Y)$, $H(Y|X)$, and maximization of mutual information $I(X;Y)$, over convex regions. When restricted to the so-called transportation polytopes (sets of distributions with fixed marginals), very simple proofs of NP-hardness are obtained for these problems because in that case they are all equivalent, and their connection to the well-known \textsc{Subset sum} and \textsc{Partition} problems is revealed. The computational intractability of the more general problems over arbitrary polytopes is then a simple consequence. Further, a simple class of polytopes is shown over which the above problems are not equivalent and their complexity differs sharply, namely, minimization of $H(X,Y)$ and $H(Y|X)$ is trivial, while minimization of $H(X|Y)$ and maximization of $I(X;Y)$ are strongly NP-hard problems. Finally, two new (pseudo)metrics on the space of discrete probability distributions are introduced, based on the so-called variation of information quantity, and NP-hardness of their computation is shown.