Semantic Variational Bayes Based on Semantic Information G Theory for Solving Latent Variables

TL;DR

This paper introduces Semantic Variational Bayes (SVB), a new method that simplifies Bayesian inference for latent variables by leveraging semantic information theory, with applications in data compression and control tasks.

Contribution

The paper proposes SVB, a novel semantic information-based variational Bayesian method that reduces computational complexity and enhances interpretability over traditional VB.

Findings

SVB converges as G/R increases in mixture models.

SVB effectively applies to data compression with error range constraints.

SVB demonstrates potential in maximum entropy control and reinforcement learning.

Abstract

The Variational Bayesian method (VB) is used to solve the probability distributions of latent variables with the minimum free energy criterion. This criterion is not easy to understand, and the computation is complex. For these reasons, this paper proposes the Semantic Variational Bayes' method (SVB). The Semantic Information Theory the author previously proposed extends the rate-distortion function R(D) to the rate-fidelity function R(G), where R is the minimum mutual information for given semantic mutual information G. SVB came from the parameter solution of R(G), where the variational and iterative methods originated from Shannon et al.'s research on the rate-distortion function. The constraint functions SVB uses include likelihood, truth, membership, similarity, and distortion functions. SVB uses the maximum information efficiency (G/R) criterion, including the maximum semantic information criterion for optimizing model parameters and the minimum mutual information criterion for optimizing the Shannon channel. For the same tasks, SVB is computationally simpler than VB. The computational experiments in the paper include 1) using a mixture model as an example to show that the mixture model converges as G/R increases; 2) demonstrating the application of SVB in data compression with a group of error ranges as the constraint; 3) illustrating how the semantic information measure and SVB can be used for maximum entropy control and reinforcement learning in control tasks with given range constraints, providing numerical evidence for balancing control's purposiveness and efficiency. Further research is needed to apply SVB to neural networks and deep learning.