The Semantic Information Method for Maximum Mutual Information and Maximum Likelihood of Tests, Estimations, and Mixture Models

TL;DR

This paper introduces the Channels' Matching (CM) algorithm, an iterative method leveraging semantic mutual information to efficiently solve maximum mutual information and likelihood problems in tests, estimations, and mixture models.

Contribution

It presents a novel iterative algorithm based on semantic mutual information and R(G) function, offering improved convergence over traditional methods like EM.

Findings

CM algorithm converges in about 5 iterations for most cases.

The CM algorithm outperforms EM in convergence speed and potential applications.

Examples demonstrate the simplicity and effectiveness of the method.

Abstract

It is very difficult to solve the Maximum Mutual Information (MMI) or Maximum Likelihood (ML) for all possible Shannon Channels or uncertain rules of choosing hypotheses, so that we have to use iterative methods. According to the Semantic Mutual Information (SMI) and R(G) function proposed by Chenguang Lu (1993) (where R(G) is an extension of information rate distortion function R(D), and G is the lower limit of the SMI), we can obtain a new iterative algorithm of solving the MMI and ML for tests, estimations, and mixture models. The SMI is defined by the average log normalized likelihood. The likelihood function is produced from the truth function and the prior by semantic Bayesian inference. A group of truth functions constitute a semantic channel. Letting the semantic channel and Shannon channel mutually match and iterate, we can obtain the Shannon channel that maximizes the Shannon mutual information and the average log likelihood. This iterative algorithm is called Channels' Matching algorithm or the CM algorithm. The convergence can be intuitively explained and proved by the R(G) function. Several iterative examples for tests, estimations, and mixture models show that the computation of the CM algorithm is simple (which can be demonstrated in excel files). For most random examples, the numbers of iterations for convergence are close to 5. For mixture models, the CM algorithm is similar to the EM algorithm; however, the CM algorithm has better convergence and more potential applications in comparison with the standard EM algorithm.