Spectral Learning for Supervised Topic Models

TL;DR

This paper introduces spectral algorithms for supervised LDA that are provably correct, efficient, and outperform existing methods in both synthetic and real-world datasets, including large-scale review data.

Contribution

It develops novel spectral algorithms for supervised LDA, providing theoretical guarantees, sample complexity bounds, and demonstrating superior empirical performance.

Findings

Spectral algorithms are provably correct and computationally efficient.

The single-phase spectral method achieves comparable or better results than state-of-the-art.

Experiments on large-scale datasets validate the practical effectiveness of the proposed methods.

Abstract

Supervised topic models simultaneously model the latent topic structure of large collections of documents and a response variable associated with each document. Existing inference methods are based on variational approximation or Monte Carlo sampling, which often suffers from the local minimum defect. Spectral methods have been applied to learn unsupervised topic models, such as latent Dirichlet allocation (LDA), with provable guarantees. This paper investigates the possibility of applying spectral methods to recover the parameters of supervised LDA (sLDA). We first present a two-stage spectral method, which recovers the parameters of LDA followed by a power update method to recover the regression model parameters. Then, we further present a single-phase spectral algorithm to jointly recover the topic distribution matrix as well as the regression weights. Our spectral algorithms are provably correct and computationally efficient. We prove a sample complexity bound for each algorithm and subsequently derive a sufficient condition for the identifiability of sLDA. Thorough experiments on synthetic and real-world datasets verify the theory and demonstrate the practical effectiveness of the spectral algorithms. In fact, our results on a large-scale review rating dataset demonstrate that our single-phase spectral algorithm alone gets comparable or even better performance than state-of-the-art methods, while previous work on spectral methods has rarely reported such promising performance.