LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

TL;DR

LexSubCon is a novel framework that enhances lexical substitution by integrating structured lexical knowledge with contextual embeddings, leading to more accurate substitute generation.

Contribution

It introduces a mix-up embedding strategy and combines lexical resources with contextual models for improved lexical substitution performance.

Findings

Outperforms state-of-the-art on LS07 and CoInCo datasets

Effective integration of lexical knowledge improves substitution accuracy

Utilizes sentence similarity modeling for better context understanding

Abstract

Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy in the creation of the input embedding of the target word through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution in the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.