On the Curious Case of $\ell_2$ norm of Sense Embeddings

TL;DR

This paper reveals that the $\, ext{l}_2$ norm of static sense embeddings encodes sense frequency information and can enhance performance in word sense disambiguation tasks.

Contribution

It extends the understanding of $\, ext{l}_2$ norm significance from word to sense embeddings and demonstrates its utility in improving sense-related NLP tasks.

Findings

$\, ext{l}_2$ norm correlates with sense frequency

Including $\, ext{l}_2$ norm improves WSD and WiC accuracy

Simple norm feature enhances sense disambiguation methods

Abstract

We show that the $\ell_2$ norm of a static sense embedding encodes information related to the frequency of that sense in the training corpus used to learn the sense embeddings. This finding can be seen as an extension of a previously known relationship for word embeddings to sense embeddings. Our experimental results show that, in spite of its simplicity, the $\ell_2$ norm of sense embeddings is a surprisingly effective feature for several word sense related tasks such as (a) most frequent sense prediction, (b) Word-in-Context (WiC), and (c) Word Sense Disambiguation (WSD). In particular, by simply including the $\ell_2$ norm of a sense embedding as a feature in a classifier, we show that we can improve WiC and WSD methods that use static sense embeddings.