Tokenisation is NP-Complete
Philip Whittington, Gregor Bachmann, Tiago Pimentel
TL;DR
This paper proves that two variants of the tokenisation problem, involving dataset compression through vocabulary creation or merge operations, are NP-complete, highlighting their computational complexity.
Contribution
The paper establishes the NP-completeness of direct and bottom-up tokenisation problems, providing a theoretical foundation for their computational difficulty.
Findings
Proves NP-completeness of direct tokenisation
Proves NP-completeness of bottom-up tokenisation
Highlights computational challenges in dataset compression
Abstract
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsEmbedded Systems Design Techniques