Nugget: Neural Agglomerative Embeddings of Text
Guanghui Qin, Benjamin Van Durme

TL;DR
Nugget introduces a dynamic token subset encoding for text, capturing variable information content and enabling larger context windows in language models, improving semantic comparison tasks.
Contribution
The paper presents Nugget, a novel method for encoding text with variable-length representations based on learned token subsets, enhancing language understanding and model context capacity.
Findings
Outperforms related approaches in semantic comparison tasks
Enables language models to condition on larger content
Segments language into meaningful units
Abstract
Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information contained in text often varies with the length of the input. We propose a solution called Nugget, which encodes language into a representation based on a dynamically selected subset of input tokens. These nuggets are learned through tasks like autoencoding and machine translation, and intuitively segment language into meaningful units. We demonstrate Nugget outperforms related approaches in tasks involving semantic comparison. Finally, we illustrate these compact units allow for expanding the contextual window of a language model (LM), suggesting new future LMs that can condition on significantly larger amounts of content.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
MethodsFocus
