LangBiTe: A Platform for Testing Bias in Large Language Models

TL;DR

LangBiTe is a testing platform designed to systematically evaluate biases in large language models by enabling customizable test scenarios and automatic bias detection, aiding developers in ethical assessment.

Contribution

This paper introduces LangBiTe, a novel platform for bias testing in LLMs that allows tailored scenarios and automated bias analysis based on user-defined ethical criteria.

Findings

Effective bias detection in LLMs demonstrated

Supports customizable and automated bias testing

Provides traceability from ethical requirements to results

Abstract

The integration of Large Language Models (LLMs) into various software applications raises concerns about their potential biases. Typically, those models are trained on a vast amount of data scrapped from forums, websites, social media and other internet sources, which may instill harmful and discriminating behavior into the model. To address this issue, we present LangBiTe, a testing platform to systematically assess the presence of biases within an LLM. LangBiTe enables development teams to tailor their test scenarios, and automatically generate and execute the test cases according to a set of user-defined ethical requirements. Each test consists of a prompt fed into the LLM and a corresponding test oracle that scrutinizes the LLM's response for the identification of biases. LangBite provides users with the bias evaluation of LLMs, and end-to-end traceability between the initial ethical requirements and the insights obtained.