Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies

TL;DR

This paper surveys current methods for identifying and mitigating bias in large language models, emphasizing the unique needs of under-represented societies and highlighting the importance of tailored approaches.

Contribution

It provides a comprehensive overview of bias mitigation techniques and explores the perspective of under-represented societies, proposing the need for adapted solutions.

Findings

Current bias mitigation methods are insufficient for under-represented societies

Existing techniques need adaptation to address societal diversity

Case studies from New Zealand illustrate tailored requirements

Abstract

The benefits and capabilities of pre-trained language models (LLMs) in current and future innovations are vital to any society. However, introducing and using LLMs comes with biases and discrimination, resulting in concerns about equality, diversity and fairness, and must be addressed. While understanding and acknowledging bias in LLMs and developing mitigation strategies are crucial, the generalised assumptions towards societal needs can result in disadvantages towards under-represented societies and indigenous populations. Furthermore, the ongoing changes to actual and proposed amendments to regulations and laws worldwide also impact research capabilities in tackling the bias problem. This research presents a comprehensive survey synthesising the current trends and limitations in techniques used for identifying and mitigating bias in LLMs, where the overview of methods for tackling bias are grouped into metrics, benchmark datasets, and mitigation strategies. The importance and novelty of this survey are that it explores the perspective of under-represented societies. We argue that current practices tackling the bias problem cannot simply be 'plugged in' to address the needs of under-represented societies. We use examples from New Zealand to present requirements for adopting existing techniques to under-represented societies.