Adversarial Contrastive Estimation
Avishek Joey Bose, Huan Ling, Yanshuai Cao

TL;DR
This paper introduces an adversarial contrastive estimation method that enhances negative sampling by learning harder negatives, leading to faster convergence and better embeddings across NLP and knowledge graph tasks.
Contribution
It proposes an adversarially learned negative sampler that adaptively finds challenging negatives, improving contrastive learning effectiveness.
Findings
Faster convergence in embedding training
Improved performance on multiple metrics
Effective across different embedding tasks
Abstract
Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in NLP employing this approach. In this work, we view contrastive learning as an abstraction of all such methods and augment the negative sampler into a mixture distribution containing an adversarially learned sampler. The resulting adaptive sampler finds harder negative examples, which forces the main model to learn a better representation of the data. We evaluate our proposal on learning word embeddings, order embeddings and knowledge graph embeddings and observe both faster convergence and improved results on multiple metrics.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Domain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning
