Lightweight Toxicity Detection in Spoken Language: A Transformer-based Approach for Edge Devices

TL;DR

This paper introduces a lightweight, transformer-based speech toxicity detection model optimized for edge devices, achieving high accuracy while significantly reducing model size and computational requirements.

Contribution

It presents the first end-to-end speech-based toxicity detection model suitable for deployment on physical edge devices using a lightweight transformer architecture.

Findings

Achieves 90.3% macro F1-score and 88% accuracy on benchmark datasets.

Quantization reduces model size by 4x and RAM usage by 3.3% with minimal accuracy loss.

Knowledge distillation decreases model size by 3.7x and inference time by 2x, with some accuracy reduction.

Abstract

Toxicity is a prevalent social behavior that involves the use of hate speech, offensive language, bullying, and abusive speech. While text-based approaches for toxicity detection are common, there is limited research on processing speech signals in the physical world. Detecting toxicity in the physical world is challenging due to the difficulty of integrating AI-capable computers into the environment. We propose a lightweight transformer model based on wav2vec2.0 and optimize it using techniques such as quantization and knowledge distillation. Our model uses multitask learning and achieves an average macro F1-score of 90.3\% and a weighted accuracy of 88\%, outperforming state-of-the-art methods on DeToxy-B and a public dataset. Our results show that quantization reduces the model size by almost 4 times and RAM usage by 3.3\%, with only a 1\% F1 score decrease. Knowledge distillation reduces the model size by 3.7 times, RAM usage by 1.9, and inference time by 2 times, but decreases accuracy by 8\%. Combining both techniques reduces the model size by 14.6 times and RAM usage by around 4.3 times, with a two-fold inference time improvement. Our compact model is the first end-to-end speech-based toxicity detection model based on a lightweight transformer model suitable for deployment in physical spaces. The results show its feasibility for toxicity detection on edge devices in real-world environments.