Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

TL;DR

This paper introduces FilBBQ, a culturally aware Filipino bias benchmark for question-answering models, assessing stereotypes like sexism and homophobia with over 10,000 prompts, and proposes a robust evaluation protocol.

Contribution

It expands the BBQ framework to the Filipino context, creating a new benchmark with improved reliability and addressing bias variability across model responses.

Findings

Models exhibit significant sexist and homophobic biases.

Bias scores vary across different response seeds.

FilBBQ is publicly available for further research.

Abstract

With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models' response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.