Segmentation-free Compositional $n$-gram Embedding

TL;DR

This paper introduces a segmentation-free embedding method that models character n-grams directly from raw, unsegmented text, effectively handling noisy corpora in languages like Chinese and Japanese.

Contribution

It presents a novel segmentation-free approach to learn embeddings for all character n-grams without relying on word boundaries or annotated resources.

Findings

Effective on noisy, unsegmented corpora

Outperforms segmentation-based methods on benchmarks

Applicable to Chinese and Japanese text

Abstract

We propose a new type of representation learning method that models words, phrases and sentences seamlessly. Our method does not depend on word segmentation and any human-annotated resources (e.g., word dictionaries), yet it is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. The main idea of our method is to ignore word boundaries completely (i.e., segmentation-free), and construct representations for all character $n$-grams in a raw corpus with embeddings of compositional sub-$n$-grams. Although the idea is simple, our experiments on various benchmarks and real-world datasets show the efficacy of our proposal.