KR-BERT: A Small-Scale Korean-Specific Language Model

TL;DR

KR-BERT is a compact Korean-specific language model trained on a smaller dataset that outperforms larger models by capturing language-specific features effectively.

Contribution

The paper introduces a Korean-specific BERT model with a tailored vocabulary and tokenizer, achieving high performance with significantly less data and resources.

Findings

KR-BERT performs comparably or better than larger models.

Customized tokenization improves language representation.

Small dataset suffices for effective Korean language modeling.

Abstract

Since the appearance of BERT, recent works including XLNet and RoBERTa utilize sentence embedding models pre-trained by large corpora and a large number of parameters. Because such models have large hardware and a huge amount of data, they take a long time to pre-train. Therefore it is important to attempt to make smaller models that perform comparatively. In this paper, we trained a Korean-specific model KR-BERT, utilizing a smaller vocabulary and dataset. Since Korean is one of the morphologically rich languages with poor resources using non-Latin alphabets, it is also important to capture language-specific linguistic phenomena that the Multilingual BERT model missed. We tested several tokenizers including our BidirectionalWordPiece Tokenizer and adjusted the minimal span of tokens for tokenization ranging from sub-character level to character-level to construct a better vocabulary for our model. With those adjustments, our KR-BERT model performed comparably and even better than other existing pre-trained models using a corpus about 1/10 of the size.