From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection

TL;DR

This paper presents a pipeline for adapting the RoBERTa language model to Vietnamese hate speech detection, significantly improving performance and achieving state-of-the-art results.

Contribution

It introduces a novel fine-tuning pipeline with techniques like layer freezing and label smoothing for Vietnamese hate speech detection.

Findings

Achieved a new state-of-the-art F1 score of 0.7221.

Demonstrated the effectiveness of the proposed fine-tuning techniques.

Significantly boosted model performance on the Vietnamese hate speech dataset.

Abstract

Natural language processing is a fast-growing field of artificial intelligence. Since the Transformer was introduced by Google in 2017, a large number of language models such as BERT, GPT, and ELMo have been inspired by this architecture. These models were trained on huge datasets and achieved state-of-the-art results on natural language understanding. However, fine-tuning a pre-trained language model on much smaller datasets for downstream tasks requires a carefully-designed pipeline to mitigate problems of the datasets such as lack of training data and imbalanced data. In this paper, we propose a pipeline to adapt the general-purpose RoBERTa language model to a specific text classification task: Vietnamese Hate Speech Detection. We first tune the PhoBERT on our dataset by re-training the model on the Masked Language Model task; then, we employ its encoder for text classification. In order to preserve pre-trained weights while learning new feature representations, we further utilize different training techniques: layer freezing, block-wise learning rate, and label smoothing. Our experiments proved that our proposed pipeline boosts the performance significantly, achieving a new state-of-the-art on Vietnamese Hate Speech Detection campaign with 0.7221 F1 score.