HateTinyLLM : Hate Speech Detection Using Tiny Large Language Models

TL;DR

HateTinyLLM introduces a fine-tuned tiny large language model framework for hate speech detection, outperforming larger pretrained models and achieving over 80% accuracy with various tiny LLMs and fine-tuning methods.

Contribution

The paper presents a novel approach using fine-tuned tiny LLMs for hate speech detection, demonstrating superior performance over existing pretrained models.

Findings

Fine-tuned HateTinyLLM outperforms pretrained mixtral-7b.

All LoRA-based models achieved over 80% accuracy.

Various tiny LLMs were effectively fine-tuned for hate speech detection.

Abstract

Hate speech encompasses verbal, written, or behavioral communication that targets derogatory or discriminatory language against individuals or groups based on sensitive characteristics. Automated hate speech detection plays a crucial role in curbing its propagation, especially across social media platforms. Various methods, including recent advancements in deep learning, have been devised to address this challenge. In this study, we introduce HateTinyLLM, a novel framework based on fine-tuned decoder-only tiny large language models (tinyLLMs) for efficient hate speech detection. Our experimental findings demonstrate that the fine-tuned HateTinyLLM outperforms the pretrained mixtral-7b model by a significant margin. We explored various tiny LLMs, including PY007/TinyLlama-1.1B-step-50K-105b, Microsoft/phi-2, and facebook/opt-1.3b, and fine-tuned them using LoRA and adapter methods. Our observations indicate that all LoRA-based fine-tuned models achieved over 80\% accuracy.