Mark my Word: A Sequence-to-Sequence Approach to Definition Modeling

TL;DR

This paper introduces a sequence-to-sequence Transformer model for generating word definitions within context, achieving state-of-the-art results and enabling end-to-end training for contextualized definition modeling.

Contribution

It formalizes definition modeling as a sequence-to-sequence task and implements an end-to-end Transformer-based approach, improving over prior methods.

Findings

Achieved state-of-the-art results in definition modeling.

Enabled end-to-end training for contextual and non-contextual definitions.

Demonstrated the effectiveness of sequence-to-sequence formalization.

Abstract

Defining words in a textual context is a useful task both for practical purposes and for gaining insight into distributed word representations. Building on the distributional hypothesis, we argue here that the most natural formalization of definition modeling is to treat it as a sequence-to-sequence task, rather than a word-to-sequence task: given an input sequence with a highlighted word, generate a contextually appropriate definition for it. We implement this approach in a Transformer-based sequence-to-sequence model. Our proposal allows to train contextualization and definition generation in an end-to-end fashion, which is a conceptual improvement over earlier works. We achieve state-of-the-art results both in contextual and non-contextual definition modeling.