Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization

TL;DR

This paper establishes the asymptotic Bayesian generalization error for Latent Dirichlet Allocation and stochastic matrix factorization, showing they share the same RLCT and have smaller errors than regular models.

Contribution

It proves the equivalence of LDA and SMF in terms of RLCT and generalization error, providing new insights into their asymptotic behavior.

Findings

LDA and SMF have the same RLCT and asymptotic generalization error.

The RLCT upper bound is smaller than half the parameter dimension.

Their Bayesian generalization errors are smaller than those of regular models.

Abstract

Latent Dirichlet allocation (LDA) is useful in document analysis, image processing, and many information systems; however, its generalization performance has been left unknown because it is a singular learning machine to which regular statistical theory can not be applied. Stochastic matrix factorization (SMF) is a restricted matrix factorization in which matrix factors are stochastic; the column of the matrix is in a simplex. SMF is being applied to image recognition and text mining. We can understand SMF as a statistical model by which a stochastic matrix of given data is represented by a product of two stochastic matrices, whose generalization performance has also been left unknown because of non-regularity. In this paper, by using an algebraic and geometric method, we show the analytic equivalence of LDA and SMF, both of which have the same real log canonical threshold (RLCT), resulting in that they asymptotically have the same Bayesian generalization error and the same log marginal likelihood. Moreover, we derive the upper bound of the RLCT and prove that it is smaller than the dimension of the parameter divided by two, hence the Bayesian generalization errors of them are smaller than those of regular statistical models.