Iterative minimization algorithm on a mixture family

TL;DR

This paper introduces a generalized iterative minimization algorithm that unifies approaches in machine learning and information theory, providing convergence guarantees and improvements for the EM algorithm and related problems.

Contribution

It generalizes a recent algorithm to unify iterative minimization methods across machine learning and information theory, with convergence analysis and practical enhancements.

Findings

Convergence theorems including approximate steps

Improved EM algorithm based on the generalized method

Application to various information theory problems

Abstract

Iterative minimization algorithms appear in various areas including machine learning, neural networks, and information theory.The em algorithm is one of the famous iterative minimization algorithms in the area of machine learning, and the Arimoto-Blahut algorithm is a typical iterative algorithm in the area of information theory.However, these two topics had been separately studied for a long time. In this paper, we generalize an algorithm that was recently proposed in the context of the Arimoto-Blahut algorithm.Then, we show various convergence theorems, one of which covers the case when each iterative step is done approximately.Also, we apply this algorithm to the target problem of the em algorithm, and propose its improvement. In addition, we apply it to other various problems in information theory.