Nugget: Neural Agglomerative Embeddings of Text

TL;DR

Nugget introduces a dynamic token subset encoding for text, capturing variable information content and enabling larger context windows in language models, improving semantic comparison tasks.

Contribution

The paper presents Nugget, a novel method for encoding text with variable-length representations based on learned token subsets, enhancing language understanding and model context capacity.

Findings

Outperforms related approaches in semantic comparison tasks

Enables language models to condition on larger content

Segments language into meaningful units

Abstract

Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information contained in text often varies with the length of the input. We propose a solution called Nugget, which encodes language into a representation based on a dynamically selected subset of input tokens. These nuggets are learned through tasks like autoencoding and machine translation, and intuitively segment language into meaningful units. We demonstrate Nugget outperforms related approaches in tasks involving semantic comparison. Finally, we illustrate these compact units allow for expanding the contextual window of a language model (LM), suggesting new future LMs that can condition on significantly larger amounts of content.