"The Dentist is an involved parent, the bartender is not": Revealing Implicit Biases in QA with Implicit BBQ

TL;DR

Implicit biases in large language models are often undetected by explicit benchmarks, and ImplicitBBQ provides a new evaluation tool to reveal these hidden biases across multiple categories.

Contribution

The paper introduces ImplicitBBQ, a benchmark for evaluating implicit biases in LLMs, extending existing fairness assessments to include implicit cues.

Findings

GPT-4o shows accuracy drops up to 7% on implicit bias prompts.

Implicit biases are prevalent in LLMs and go undetected by explicit benchmarks.

ImplicitBBQ enables more nuanced fairness evaluations in NLP.

Abstract

Existing benchmarks evaluating biases in large language models (LLMs) primarily rely on explicit cues, declaring protected attributes like religion, race, gender by name. However, real-world interactions often contain implicit biases, inferred subtly through names, cultural cues, or traits. This critical oversight creates a significant blind spot in fairness evaluation. We introduce ImplicitBBQ, a benchmark extending the Bias Benchmark for QA (BBQ) with implicitly cued protected attributes across 6 categories. Our evaluation of GPT-4o on ImplicitBBQ illustrates troubling performance disparity from explicit BBQ prompts, with accuracy declining up to 7% in the "sexual orientation" subcategory and consistent decline located across most other categories. This indicates that current LLMs contain implicit biases undetected by explicit benchmarks. ImplicitBBQ offers a crucial tool for nuanced fairness evaluation in NLP.