mcBERT: Momentum Contrastive Learning with BERT for Zero-Shot Slot Filling

TL;DR

mcBERT introduces a momentum contrastive learning approach with BERT for zero-shot slot filling, achieving state-of-the-art results by learning more generalized and reliable representations for unseen domains.

Contribution

The paper proposes mcBERT, a novel zero-shot slot filling model that leverages momentum contrastive learning with BERT to improve generalization and performance.

Findings

Outperforms previous models on SNIPS benchmark

Achieves new state-of-the-art results

Each component of mcBERT contributes to performance gains

Abstract

Zero-shot slot filling has received considerable attention to cope with the problem of limited available data for the target domain. One of the important factors in zero-shot learning is to make the model learn generalized and reliable representations. For this purpose, we present mcBERT, which stands for momentum contrastive learning with BERT, to develop a robust zero-shot slot filling model. mcBERT uses BERT to initialize the two encoders, the query encoder and key encoder, and is trained by applying momentum contrastive learning. Our experimental results on the SNIPS benchmark show that mcBERT substantially outperforms the previous models, recording a new state-of-the-art. Besides, we also show that each component composing mcBERT contributes to the performance improvement.