Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection

TL;DR

This paper proposes a diversity-based prioritization method for metamorphic testing of large language models to efficiently detect fairness issues, significantly improving fault detection rates and reducing testing time.

Contribution

It introduces a novel diversity-based approach to prioritize metamorphic relations, enhancing fairness testing efficiency in large language models.

Findings

Improves fault detection rate by 22% over random prioritization

Reduces time to first failure by 15%

Performs within 5% of fault-based prioritization in effectiveness

Abstract

Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.