R-grams: Unsupervised Learning of Semantic Units in Natural Language

TL;DR

This paper explores data-driven segmentation methods to identify semantic units called r-grams, extending beyond subword units, and evaluates their effectiveness in language modeling and embedding tasks across monolingual and multilingual contexts.

Contribution

It introduces the concept of r-grams as a general segmentation approach and demonstrates their properties and utility in language-invariant embedding techniques.

Findings

R-grams influence token frequency distributions.

Effective in monolingual and multilingual embedding tasks.

Qualitative examples show language-invariant segmentation viability.

Abstract

This paper investigates data-driven segmentation using Re-Pair or Byte Pair Encoding-techniques. In contrast to previous work which has primarily been focused on subword units for machine translation, we are interested in the general properties of such segments above the word level. We call these segments r-grams, and discuss their properties and the effect they have on the token frequency distribution. The proposed approach is evaluated by demonstrating its viability in embedding techniques, both in monolingual and multilingual test settings. We also provide a number of qualitative examples of the proposed methodology, demonstrating its viability as a language-invariant segmentation procedure.