Entry-Wise Eigenvector Analysis and Improved Rates for Topic Modeling on Short Documents

TL;DR

This paper establishes the optimal estimation rate for topic models on short documents, introduces entry-wise bounds for empirical singular vectors, and improves spectral algorithm performance to match the theoretical limit.

Contribution

It provides new entry-wise large-deviation bounds and demonstrates that the optimal rate for topic modeling remains the same in short-document scenarios.

Findings

Optimal rate for short documents is 

Improved spectral algorithm matches the minimax lower bound in short-document case

Entry-wise bounds enhance understanding of empirical singular vectors in topic models

Abstract

Topic modeling is a widely utilized tool in text analysis. We investigate the optimal rate for estimating a topic model. Specifically, we consider a scenario with $n$ documents, a vocabulary of size $p$, and document lengths at the order $N$. When $N\geq c\cdot p$, referred to as the long-document case, the optimal rate is established in the literature at $\sqrt{p/(Nn)}$. However, when $N=o(p)$, referred to as the short-document case, the optimal rate remains unknown. In this paper, we first provide new entry-wise large-deviation bounds for the empirical singular vectors of a topic model. We then apply these bounds to improve the error rate of a spectral algorithm, Topic-SCORE. Finally, by comparing the improved error rate with the minimax lower bound, we conclude that the optimal rate is still $\sqrt{p/(Nn)}$ in the short-document case.