LegalTurk Optimized BERT for Multi-Label Text Classification and NER

TL;DR

This paper introduces a modified pre-training approach for BERT tailored to the legal Turkish domain, improving performance in NER and multi-label classification tasks even with smaller training corpora.

Contribution

The study presents a novel pre-training method combining diverse masking strategies specifically for legal Turkish BERT models, enhancing downstream task performance.

Findings

Modified pre-training improves NER and classification accuracy

Models trained on smaller corpora outperform original BERT

Proposed approach is competitive with BERTurk despite less data

Abstract

The introduction of the Transformer neural network, along with techniques like self-supervised pre-training and transfer learning, has paved the way for advanced models like BERT. Despite BERT's impressive performance, opportunities for further enhancement exist. To our knowledge, most efforts are focusing on improving BERT's performance in English and in general domains, with no study specifically addressing the legal Turkish domain. Our study is primarily dedicated to enhancing the BERT model within the legal Turkish domain through modifications in the pre-training phase. In this work, we introduce our innovative modified pre-training approach by combining diverse masking strategies. In the fine-tuning task, we focus on two essential downstream tasks in the legal domain: name entity recognition and multi-label text classification. To evaluate our modified pre-training approach, we fine-tuned all customized models alongside the original BERT models to compare their performance. Our modified approach demonstrated significant improvements in both NER and multi-label text classification tasks compared to the original BERT model. Finally, to showcase the impact of our proposed models, we trained our best models with different corpus sizes and compared them with BERTurk models. The experimental results demonstrate that our innovative approach, despite being pre-trained on a smaller corpus, competes with BERTurk.