Optimizing Korean-Centric LLMs via Token Pruning

TL;DR

This paper systematically benchmarks multilingual LLMs with token pruning for Korean NLP, showing improved stability, translation performance, and memory efficiency, with architecture-dependent effects on instruction following.

Contribution

It demonstrates that token pruning effectively optimizes multilingual LLMs for Korean tasks, enhancing stability and efficiency while maintaining performance.

Findings

Token pruning improves generation stability by reducing language confusion.

Machine translation performance on Korean tasks is often enhanced by token pruning.

Vocabulary reduction leads to significant memory savings with modest latency gains.

Abstract

This paper presents a systematic benchmark of state-of-the-art multilingual large language models (LLMs) adapted via token pruning - a compression technique that eliminates tokens and embedding parameters corresponding to languages irrelevant to the target application. Focusing on Korean-centric natural language processing (NLP) tasks, we evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations: Original, English-Korean (EnKo), and English-Korean-Chinese (EnKoZh). Performance is assessed using established benchmarks for general aptitude, cultural literacy, instruction following, and machine translation. Our findings indicate that token pruning significantly improves generation stability by eliminating language confusion, and in the case of machine translation, frequently enhances performance on Korean-specific tasks. While instruction-following capabilities display architecture-dependent variance linked to latent cross-lingual representations, the significant reduction in vocabulary size validates token pruning as a highly effective optimization strategy for memory-constrained, domain-specific deployments, despite modest gains in inference latency.