Korean-English Machine Translation with Multiple Tokenization Strategy

TL;DR

This study investigates how different tokenization strategies impact Korean-English machine translation performance, finding that BPE for Korean and morpheme for English yields the best BLEU score.

Contribution

It systematically compares multiple tokenization methods for Korean-English translation using Transformer models, highlighting the optimal combination for improved accuracy.

Findings

BPE tokenization on Korean improves translation quality.

Morpheme tokenization on English enhances BLEU scores.

The best model achieved a BLEU score of 35.73.

Abstract

This work was conducted to find out how tokenization methods affect the training results of machine translation models. In this work, alphabet tokenization, morpheme tokenization, and BPE tokenization were applied to Korean as the source language and English as the target language respectively, and the comparison experiment was conducted by repeating 50,000 epochs of each 9 models using the Transformer neural network. As a result of measuring the BLEU scores of the experimental models, the model that applied BPE tokenization to Korean and morpheme tokenization to English recorded 35.73, showing the best performance.