Candidates vs. Noises Estimation for Large Multi-Class Classification Problem

TL;DR

This paper introduces CANE, a scalable and efficient method for large multi-class classification that improves accuracy and speed by selecting candidate classes and sampling the rest, with applications to neural language models.

Contribution

The paper presents CANE, a novel candidate vs. noises estimation approach that is consistent, computationally efficient, and achieves low variance, outperforming existing methods in large-scale classification.

Findings

CANE achieves better prediction accuracy than NCE and state-of-the-art tree classifiers.

CANE provides significant speedup over traditional O(K) methods.

CANE is consistent and has low statistical variance.

Abstract

This paper proposes a method for multi-class classification problems, where the number of classes K is large. The method, referred to as Candidates vs. Noises Estimation (CANE), selects a small subset of candidate classes and samples the remaining classes. We show that CANE is always consistent and computationally efficient. Moreover, the resulting estimator has low statistical variance approaching that of the maximum likelihood estimator, when the observed label belongs to the selected candidates with high probability. In practice, we use a tree structure with leaves as classes to promote fast beam search for candidate selection. We further apply the CANE method to estimate word probabilities in learning large neural language models. Extensive experimental results show that CANE achieves better prediction accuracy over the Noise-Contrastive Estimation (NCE), its variants and a number of the state-of-the-art tree classifiers, while it gains significant speedup compared to standard O(K) methods.