Efficient Learning for Undirected Topic Models

TL;DR

This paper introduces a novel estimator based on Noise Contrastive Estimate to improve the efficiency of learning in undirected topic models, specifically the Replicated Softmax model, leading to faster training and high accuracy.

Contribution

The paper extends Noise Contrastive Estimate for variable-length and weighted documents, significantly enhancing learning speed and accuracy in undirected topic models.

Findings

Achieves faster training compared to Contrastive Divergence

Maintains high accuracy in document retrieval and classification

Effective for documents of varying lengths and weights

Abstract

Replicated Softmax model, a well-known undirected topic model, is powerful in extracting semantic representations of documents. Traditional learning strategies such as Contrastive Divergence are very inefficient. This paper provides a novel estimator to speed up the learning based on Noise Contrastive Estimate, extended for documents of variant lengths and weighted inputs. Experiments on two benchmarks show that the new estimator achieves great learning efficiency and high accuracy on document retrieval and classification.