Efficient Hate Speech Detection: A Three-Layer LoRA-Tuned BERTweet Framework

TL;DR

This paper presents a resource-efficient hate speech detection framework that combines rule-based filtering with a LoRA-tuned BERTweet model, achieving high accuracy with minimal computational resources suitable for real-time applications.

Contribution

It introduces a three-layer framework integrating rule-based filtering and LoRA tuning of BERTweet, significantly reducing model size and training time while maintaining competitive performance.

Findings

Achieves 0.85 macro F1 score, 94% of state-of-the-art performance.

Uses only 1.87M trainable parameters, 1.37% of full fine-tuning.

Trains in approximately 2 hours on a single T4 GPU.

Abstract

This paper addresses the critical challenge of developing computationally efficient hate speech detection systems that maintain competitive performance while being practical for real-time deployment. We propose a novel three-layer framework that combines rule-based pre-filtering with a parameter-efficient LoRA-tuned BERTweet model and continuous learning capabilities. Our approach achieves 0.85 macro F1 score - representing 94% of the performance of state-of-the-art large language models like SafePhi (Phi-4 based) while using a base model that is 100x smaller (134M vs 14B parameters). Compared to traditional BERT-based approaches with similar computational requirements, our method demonstrates superior performance through strategic dataset unification and optimized fine-tuning. The system requires only 1.87M trainable parameters (1.37% of full fine-tuning) and trains in approximately 2 hours on a single T4 GPU, making robust hate speech detection accessible in resource-constrained environments while maintaining competitive accuracy for real-world deployment.