Improving Interpretability of Word Embeddings by Generating Definition and Usage

TL;DR

This paper introduces a novel framework for generating natural language definitions and usage examples from word embeddings, enhancing interpretability and understanding of semantic representations.

Contribution

It proposes a multi-task learning approach combining definition and usage modeling, achieving state-of-the-art results and improving interpretability of word embeddings.

Findings

Single-task model achieves state-of-the-art in definition modeling.

Multi-task learning improves performance on both tasks.

Models generate reasonable, context-dependent definitions and usage examples.

Abstract

Word embeddings are substantially successful in capturing semantic relations among words. However, these lexical semantics are difficult to be interpreted. Definition modeling provides a more intuitive way to evaluate embeddings by utilizing them to generate natural language definitions of corresponding words. This task is of great significance for practical application and in-depth understanding of word representations. We propose a novel framework for definition modeling, which can generate reasonable and understandable context-dependent definitions. Moreover, we introduce usage modeling and study whether it is possible to utilize embeddings to generate example sentences of words. These ways are a more direct and explicit expression of embedding's semantics for better interpretability. We extend the single task model to multi-task setting and investigate several joint multi-task models to combine usage modeling and definition modeling together. Experimental results on existing Oxford dataset and a new collected Oxford-2019 dataset show that our single-task model achieves the state-of-the-art result in definition modeling and the multi-task learning methods are helpful for two tasks to improve the performance.