Neural Token Segmentation for High Token-Internal Complexity

TL;DR

This paper introduces a neural token segmentation model that combines contextualized token representations with character-level decoding, significantly improving segmentation accuracy for complex languages like Hebrew and Arabic, and enhancing downstream NLP tasks.

Contribution

The paper presents a novel neural segmentation approach that effectively handles high token-internal complexity by integrating contextualized vectors with character-level decoding, outperforming previous methods.

Findings

Substantial improvements in Hebrew and Arabic segmentation accuracy.

Enhanced downstream NLP task performance with the proposed segmentation pipeline.

Superiority of segmentation-first pipeline over joint segmentation and labeling.

Abstract

Tokenizing raw texts into word units is an essential pre-processing step for critical tasks in the NLP pipeline such as tagging, parsing, named entity recognition, and more. For most languages, this tokenization step straightforward. However, for languages with high token-internal complexity, further token-to-word segmentation is required. Previous canonical segmentation studies were based on character-level frameworks, with no contextualised representation involved. Contextualized vectors a la BERT show remarkable results in many applications, but were not shown to improve performance on linguistic segmentation per se. Here we propose a novel neural segmentation model which combines the best of both worlds, contextualised token representation and char-level decoding, which is particularly effective for languages with high token-internal complexity and extreme morphological ambiguity. Our model shows substantial improvements in segmentation accuracy on Hebrew and Arabic compared to the state-of-the-art, and leads to further improvements on downstream tasks such as Part-of-Speech Tagging, Dependency Parsing and Named-Entity Recognition, over existing pipelines. When comparing our segmentation-first pipeline with joint segmentation and labeling in the same settings, we show that, contrary to pre-neural studies, the pipeline performance is superior.