KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding

TL;DR

KoreALBERT is a lightweight, monolingual Korean language model that introduces a new training objective, outperforming multilingual models on various Korean NLP tasks despite having fewer parameters.

Contribution

The paper develops KoreALBERT, a Korean-specific ALBERT model with a novel Word Order Prediction objective, enhancing Korean NLP performance with fewer parameters.

Findings

Outperforms BERT on 6 Korean NLP tasks

Uses fewer parameters, enabling quicker training

Improves multi-sentence encoding in Korean NLP

Abstract

A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional representation learning for natural languages. Due to the lack of pretrained ALBERT models for Korean language, the best available practice is the multilingual model or resorting back to the any other BERT-based model. In this paper, we develop and pretrain KoreALBERT, a monolingual ALBERT model specifically for Korean language understanding. We introduce a new training objective, namely Word Order Prediction (WOP), and use alongside the existing MLM and SOP criteria to the same architecture and model parameters. Despite having significantly fewer model parameters (thus, quicker to train), our pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU tasks. Consistent with the empirical results in English by Lan et al., KoreALBERT seems to improve downstream task performance involving multi-sentence encoding for Korean language. The pretrained KoreALBERT is publicly available to encourage research and application development for Korean NLP.