KoBBQ: Korean Bias Benchmark for Question Answering

TL;DR

KoBBQ is a culturally adapted Korean social bias benchmark for question answering, developed through a framework that localizes and extends the existing BBQ dataset to reflect Korean social stereotypes and biases.

Contribution

The paper introduces KoBBQ, a Korean bias benchmark dataset, and a framework for culturally adapting social bias datasets for question answering models.

Findings

KoBBQ contains 76,048 samples across 12 bias categories.

Multilingual LMs show different bias levels on KoBBQ compared to translated BBQ.

Cultural adaptation improves bias detection accuracy in Korean language models.

Abstract

The Bias Benchmark for Question Answering (BBQ) is designed to evaluate social biases of language models (LMs), but it is not simple to adapt this benchmark to cultural contexts other than the US because social biases depend heavily on the cultural context. In this paper, we present KoBBQ, a Korean bias benchmark dataset, and we propose a general framework that addresses considerations for cultural adaptation of a dataset. Our framework includes partitioning the BBQ dataset into three classes--Simply-Transferred (can be used directly after cultural translation), Target-Modified (requires localization in target groups), and Sample-Removed (does not fit Korean culture)-- and adding four new categories of bias specific to Korean culture. We conduct a large-scale survey to collect and validate the social biases and the targets of the biases that reflect the stereotypes in Korean culture. The resulting KoBBQ dataset comprises 268 templates and 76,048 samples across 12 categories of social bias. We use KoBBQ to measure the accuracy and bias scores of several state-of-the-art multilingual LMs. The results clearly show differences in the bias of LMs as measured by KoBBQ and a machine-translated version of BBQ, demonstrating the need for and utility of a well-constructed, culturally-aware social bias benchmark.