Combining Contrastive Learning and Knowledge Graph Embeddings to develop medical word embeddings for the Italian language

TL;DR

This paper presents a novel approach combining contrastive learning and knowledge graph embeddings to improve Italian medical word embeddings, enhancing semantic similarity accuracy with less data.

Contribution

It introduces a new method integrating CL and KGE for Italian medical embeddings, addressing language-specific data scarcity and domain adaptation.

Findings

Improved semantic similarity between medical terms.

Significant performance increase over initial models.

Achieved results with less data than state-of-the-art models.

Abstract

Word embeddings play a significant role in today's Natural Language Processing tasks and applications. While pre-trained models may be directly employed and integrated into existing pipelines, they are often fine-tuned to better fit with specific languages or domains. In this paper, we attempt to improve available embeddings in the uncovered niche of the Italian medical domain through the combination of Contrastive Learning (CL) and Knowledge Graph Embedding (KGE). The main objective is to improve the accuracy of semantic similarity between medical terms, which is also used as an evaluation task. Since the Italian language lacks medical texts and controlled vocabularies, we have developed a specific solution by combining preexisting CL methods (multi-similarity loss, contextualization, dynamic sampling) and the integration of KGEs, creating a new variant of the loss. Although without having outperformed the state-of-the-art, represented by multilingual models, the obtained results are encouraging, providing a significant leap in performance compared to the starting model, while using a significantly lower amount of data.