Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models

TL;DR

This paper develops and evaluates small to medium-sized Turkish BERT models trained on diverse data, demonstrating their effectiveness across multiple NLP tasks with high efficiency and robustness.

Contribution

Introduces and assesses new tiny to medium-sized Turkish BERT models, filling a research gap for less-resourced languages with comprehensive evaluation.

Findings

Models perform well on various NLP tasks.

Small models achieve competitive results.

Efficient and faster than larger counterparts.

Abstract

This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models, aiming to bridge the research gap in less-resourced languages. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification. Despite their smaller size, our models exhibited robust performance, including zero-shot task, while ensuring computational efficiency and faster execution times. Our findings provide valuable insights into the development and application of smaller language models, especially in the context of the Turkish language.