On the Connection Between Non-negative Matrix Factorization and Latent Dirichlet Allocation

TL;DR

This paper establishes a theoretical connection between non-negative matrix factorization (NMF) with specific constraints and latent Dirichlet allocation (LDA), showing their equivalence under certain conditions and clarifying the role of normalization and penalties.

Contribution

It demonstrates that NMF with  normalization and a Dirichlet prior is equivalent to LDA, and introduces joint update algorithms that clarify the relationship between these models.

Findings

NMF with  normalization and Dirichlet prior is equivalent to LDA.

Joint multiplicative updates enable a probabilistic interpretation of NMF.

Lasso penalty with normalization does not induce sparsity in the matrices.

Abstract

Non-negative matrix factorization with the generalized Kullback-Leibler divergence (NMF) and latent Dirichlet allocation (LDA) are two popular approaches for dimensionality reduction of non-negative data. Here, we show that NMF with $\ell_1$ normalization constraints on the columns of both matrices of the decomposition and a Dirichlet prior on the columns of one matrix is equivalent to LDA. To show this, we demonstrate that explicitly accounting for the scaling ambiguity of NMF by adding $\ell_1$ normalization constraints to the optimization problem allows a joint update of both matrices in the widely used multiplicative updates (MU) algorithm. When both of the matrices are normalized, the joint MU algorithm leads to probabilistic latent semantic analysis (PLSA), which is LDA without a Dirichlet prior. Our approach of deriving joint updates for NMF also reveals that a Lasso penalty on one matrix together with an $\ell_1$ normalization constraint on the other matrix is insufficient to induce any sparsity.