Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management

TL;DR

This paper introduces Q-Pain, a dataset designed to evaluate social biases in medical question answering systems related to pain management, highlighting biases in AI treatment recommendations across demographic groups.

Contribution

The paper presents a novel dataset and framework for measuring social bias in medical QA systems, specifically in pain management, and demonstrates its application on GPT-2 and GPT-3.

Findings

Significant treatment disparities across race-gender groups in AI systems.

AI models show biases that could impact clinical decision-making.

Q-Pain helps identify and quantify biases in medical NLP applications.

Abstract

Recent advances in Natural Language Processing (NLP), and specifically automated Question Answering (QA) systems, have demonstrated both impressive linguistic fluency and a pernicious tendency to reflect social biases. In this study, we introduce Q-Pain, a dataset for assessing bias in medical QA in the context of pain management, one of the most challenging forms of clinical decision-making. Along with the dataset, we propose a new, rigorous framework, including a sample experimental design, to measure the potential biases present when making treatment decisions. We demonstrate its use by assessing two reference Question-Answering systems, GPT-2 and GPT-3, and find statistically significant differences in treatment between intersectional race-gender subgroups, thus reaffirming the risks posed by AI in medical settings, and the need for datasets like ours to ensure safety before medical AI applications are deployed.