One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities

TL;DR

This paper introduces a scalable approximation to softmax probabilities using a lower bound expressed as a product over pairwise probabilities, enabling efficient large-scale inference in classification tasks.

Contribution

The authors propose a novel lower bound approximation to softmax that is computationally efficient and suitable for stochastic optimization in large-scale settings.

Findings

The approximation provides a rigorous lower bound on softmax probabilities.

It enables doubly stochastic estimation by subsampling instances and classes.

Experimental results demonstrate its effectiveness in classification problems.

Abstract

The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference because of the high cost associated with computing the normalizing constant. Here, we introduce an efficient approximation to softmax probabilities which takes the form of a rigorous lower bound on the exact probability. This bound is expressed as a product over pairwise probabilities and it leads to scalable estimation based on stochastic optimization. It allows us to perform doubly stochastic estimation by subsampling both training instances and class labels. We show that the new bound has interesting theoretical properties and we demonstrate its use in classification problems.