Bita: A Conversational Assistant for Fairness Testing

TL;DR

Bita is a conversational AI assistant that helps software testers identify and evaluate bias in AI systems by providing structured guidance and leveraging fairness literature, making fairness testing more accessible and systematic.

Contribution

We introduce Bita, a novel conversational assistant that integrates language models with fairness literature to support practical, systematic fairness testing in real-world AI applications.

Findings

Bita effectively supports fairness testing tasks on real-world AI systems.

It provides structured, reproducible evidence of bias detection and mitigation.

Bita enhances accessibility of fairness testing for practitioners.

Abstract

Bias in AI systems can lead to unfair and discriminatory outcomes, especially when left untested before deployment. Although fairness testing aims to identify and mitigate such bias, existing tools are often difficult to use, requiring advanced expertise and offering limited support for real-world workflows. To address this, we introduce Bita, a conversational assistant designed to help software testers detect potential sources of bias, evaluate test plans through a fairness lens, and generate fairness-oriented exploratory testing charters. Bita integrates a large language model with retrieval-augmented generation, grounding its responses in curated fairness literature. Our validation demonstrates how Bita supports fairness testing tasks on real-world AI systems, providing structured, reproducible evidence of its utility. In summary, our work contributes a practical tool that operationalizes fairness testing in a way that is accessible, systematic, and directly applicable to industrial practice.