Personalisation or Prejudice? Addressing Geographic Bias in Hate Speech Detection using Debias Tuning in Large Language Models

TL;DR

This paper investigates how large language models' personalisation features influence hate speech detection, revealing biases and proposing a debiasing fine-tuning method to improve fairness and accuracy across different contexts.

Contribution

It introduces a novel debias tuning approach to mitigate geographic bias in LLMs' hate speech detection, enhancing model fairness across personalisation scenarios.

Findings

Context personalisation affects hate speech responses significantly.

Debias tuning reduces geographic bias in LLMs.

Refined models perform better in diverse personalisation contexts.

Abstract

Commercial Large Language Models (LLMs) have recently incorporated memory features to deliver personalised responses. This memory retains details such as user demographics and individual characteristics, allowing LLMs to adjust their behaviour based on personal information. However, the impact of integrating personalised information into the context has not been thoroughly assessed, leading to questions about its influence on LLM behaviour. Personalisation can be challenging, particularly with sensitive topics. In this paper, we examine various state-of-the-art LLMs to understand their behaviour in different personalisation scenarios, specifically focusing on hate speech. We prompt the models to assume country-specific personas and use different languages for hate speech detection. Our findings reveal that context personalisation significantly influences LLMs' responses in this sensitive area. To mitigate these unwanted biases, we fine-tune the LLMs by penalising inconsistent hate speech classifications made with and without country or language-specific context. The refined models demonstrate improved performance in both personalised contexts and when no context is provided.