Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

TL;DR

This paper investigates the limits of unsupervised word segmentation using Large Language Models (LLMs), demonstrating their ability to follow prompts for segmentation, and introduces a novel hybrid method called LLACA that combines LLM insights with Aho-Corasick automata for improved performance.

Contribution

The paper proposes a new framework to evaluate LLMs' semantic understanding through word segmentation and introduces LLACA, a hybrid unsupervised segmentation method combining LLMs with Aho-Corasick automata.

Findings

LLMs can follow prompts to segment raw text into words.

Model size correlates with segmentation performance across languages.

LLACA improves segmentation accuracy over traditional methods.

Abstract

Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA ($\textbf{L}$arge $\textbf{L}$anguage Model-Inspired $\textbf{A}$ho-$\textbf{C}$orasick $\textbf{A}$utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic $n$-gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA