# Topic Modeling via Full Dependence Mixtures

## TL;DR

This paper introduces the Full Dependence Mixture model for scalable topic modeling that directly learns from co-occurrence data using stochastic optimization and GPU acceleration, enabling efficient analysis of large datasets.

## Contribution

The paper presents a novel FDM model for topic modeling based on second moments and develops a stochastic optimization method that leverages GPU architecture for large-scale data.

## Key findings

- Performs comparably or better than standard benchmarks on large datasets
- Scales efficiently with dataset size due to moment-based approach
- Utilizes GPU acceleration for faster computation

## Abstract

In this paper we introduce a new approach to topic modelling that scales to large datasets by using a compact representation of the data and by leveraging the GPU architecture. In this approach, topics are learned directly from the co-occurrence data of the corpus. In particular, we introduce a novel mixture model which we term the Full Dependence Mixture (FDM) model. FDMs model second moment under general generative assumptions on the data. While there is previous work on topic modeling using second moments, we develop a direct stochastic optimization procedure for fitting an FDM with a single Kullback Leibler objective. Moment methods in general have the benefit that an iteration no longer needs to scale with the size of the corpus. Our approach allows us to leverage standard optimizers and GPUs for the problem of topic modeling. In particular, we evaluate the approach on two large datasets, NeurIPS papers and a Twitter corpus, with a large number of topics, and show that the approach performs comparably or better than the the standard benchmarks.