Segmenting Natural Language Sentences via Lexical Unit Analysis

TL;DR

This paper introduces Lexical Unit Analysis (LUA), a versatile and efficient framework for sequence segmentation in natural language processing, achieving state-of-the-art results across multiple tasks and datasets.

Contribution

LUA is a novel segmentation framework that guarantees valid outputs, enables globally optimal training, and operates efficiently with linear time complexity.

Findings

Achieved state-of-the-art performance on 13 out of 15 datasets.

Significantly improved F1 scores for long-length segment identification.

Demonstrated versatility across five different NLP tasks.

Abstract

In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.