Understanding Hard Negatives in Noise Contrastive Estimation

TL;DR

This paper provides a theoretical understanding of the effectiveness of hard negatives in noise contrastive estimation, demonstrating how their use reduces bias and improves zero-shot entity linking performance.

Contribution

It introduces analytical tools to justify the use of hard negatives and unifies different architectures through a general score function, advancing noise contrastive estimation theory.

Findings

Setting the negative distribution to the model distribution reduces bias.

Hard negatives combined with a unified score function improve zero-shot entity linking.

Theoretical and empirical evidence supports the bias reduction with hard negatives.

Abstract

The choice of negative examples is important in noise contrastive estimation. Recent works find that hard negatives -- highest-scoring incorrect examples under the model -- are effective in practice, but they are used without a formal justification. We develop analytical tools to understand the role of hard negatives. Specifically, we view the contrastive loss as a biased estimator of the gradient of the cross-entropy loss, and show both theoretically and empirically that setting the negative distribution to be the model distribution results in bias reduction. We also derive a general form of the score function that unifies various architectures used in text retrieval. By combining hard negatives with appropriate score functions, we obtain strong results on the challenging task of zero-shot entity linking.