SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation

TL;DR

SimCLAD introduces a contrastive learning framework that improves acronym disambiguation by enhancing phrase representations, outperforming existing methods in scientific domain tasks.

Contribution

The paper proposes a novel contrastive pre-training approach that refines pre-trained language models for better acronym disambiguation in scientific texts.

Findings

Outperforms state-of-the-art methods on scientific acronym disambiguation

Enhances model generalization through contrastive pre-training

Improves phrase-level discrimination in vector space

Abstract

Acronym disambiguation means finding the correct meaning of an ambiguous acronym from the dictionary in a given sentence, which is one of the key points for scientific document understanding (SDU@AAAI-22). Recently, many attempts have tried to solve this problem via fine-tuning the pre-trained masked language models (MLMs) in order to obtain a better acronym representation. However, the acronym meaning is varied under different contexts, whose corresponding phrase representation mapped in different directions lacks discrimination in the entire vector space. Thus, the original representations of the pre-trained MLMs are not ideal for the acronym disambiguation task. In this paper, we propose a Simple framework for Contrastive Learning of Acronym Disambiguation (SimCLAD) method to better understand the acronym meanings. Specifically, we design a continual contrastive pre-training method that enhances the pre-trained model's generalization ability by learning the phrase-level contrastive distributions between true meaning and ambiguous phrases. The results on the acronym disambiguation of the scientific domain in English show that the proposed method outperforms all other competitive state-of-the-art (SOTA) methods.