Kernel Topic Models

TL;DR

Kernel Topic Models extend traditional topic modeling by incorporating Gaussian processes, enabling the modeling of complex structures like temporal and spatial relationships between documents.

Contribution

This paper introduces kernel topic models with an efficient Laplace approximation for inference, linking them to Gaussian process latent variable models.

Findings

Enables modeling of structured document relationships

Provides an efficient approximate inference algorithm

Unifies topic models with Gaussian process frameworks

Abstract

Latent Dirichlet Allocation models discrete data as a mixture of discrete distributions, using Dirichlet beliefs over the mixture weights. We study a variation of this concept, in which the documents' mixture weight beliefs are replaced with squashed Gaussian distributions. This allows documents to be associated with elements of a Hilbert space, admitting kernel topic models (KTM), modelling temporal, spatial, hierarchical, social and other structure between documents. The main challenge is efficient approximate inference on the latent Gaussian. We present an approximate algorithm cast around a Laplace approximation in a transformed basis. The KTM can also be interpreted as a type of Gaussian process latent variable model, or as a topic model conditional on document features, uncovering links between earlier work in these areas.