ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations

TL;DR

ViConBERT is a novel Vietnamese contextualized embedding model that leverages contrastive learning and gloss distillation, significantly improving semantic understanding tasks like WSD and similarity measurement.

Contribution

The paper introduces ViConBERT, the first Vietnamese contextualized embedding model combining contrastive learning and gloss-based distillation for better semantic representations.

Findings

Outperforms baselines on WSD with F1=0.87

Achieves AP=0.88 on ViCon similarity task

Spearman's rho=0.60 on ViSim-400

Abstract

Recent advances in contextualized word embeddings have greatly improved semantic tasks such as Word Sense Disambiguation (WSD) and contextual similarity, but most progress has been limited to high-resource languages like English. Vietnamese, in contrast, still lacks robust models and evaluation resources for fine-grained semantic understanding. In this paper, we present ViConBERT, a novel framework for learning Vietnamese contextualized embeddings that integrates contrastive learning (SimCLR) and gloss-based distillation to better capture word meaning. We also introduce ViConWSD, the first large-scale synthetic dataset for evaluating semantic understanding in Vietnamese, covering both WSD and contextual similarity. Experimental results show that ViConBERT outperforms strong baselines on WSD (F1 = 0.87) and achieves competitive performance on ViCon (AP = 0.88) and ViSim-400 (Spearman's rho = 0.60), demonstrating its effectiveness in modeling both discrete senses and graded semantic relations. Our code, models, and data are available at https://github.com/tkhangg0910/ViConBERT