E$^2$M: Double Bounded $\alpha$-Divergence Optimization for Tensor-based Discrete Density Estimation

TL;DR

This paper introduces E$^2$M, a generalized EM algorithm for tensor-based discrete density estimation that overcomes analytical challenges of $oldsymbol{ extalpha}$-divergence, enabling flexible low-rank modeling and effective learning.

Contribution

It proposes a novel EM-based optimization method that relaxes $oldsymbol{ extalpha}$-divergence challenges, allowing closed-form updates for various tensor low-rank structures.

Findings

Effective in classification tasks

Improves density estimation accuracy

Supports multiple low-rank tensor formats

Abstract

Tensor-based discrete density estimation requires flexible modeling and proper divergence criteria to enable effective learning; however, traditional approaches using $\alpha$-divergence face analytical challenges due to the $\alpha$-power terms in the objective function, which hinder the derivation of closed-form update rules. We present a generalization of the expectation-maximization (EM) algorithm, called E$^2$M algorithm. It circumvents this issue by first relaxing the optimization into minimization of a surrogate objective based on the Kullback-Leibler (KL) divergence, which is tractable via the standard EM algorithm, and subsequently applying a tensor many-body approximation in the M-step to enable simultaneous closed-form updates of all parameters. Our approach offers flexible modeling for not only a variety of low-rank structures, including the CP, Tucker, and Tensor Train formats, but also their mixtures, thus allowing us to leverage the strengths of different low-rank structures. We demonstrate the effectiveness of our approach in classification and density estimation tasks.