Identifying Fairness Issues in Automatically Generated Testing Content

TL;DR

This paper investigates fairness issues in AI-generated test content, focusing on identifying biased or inappropriate material that could unfairly affect test scores, and evaluates various classification methods to improve detection.

Contribution

It introduces a dataset of annotated test content for fairness issues and compares multiple classification approaches, highlighting the effectiveness of prompt self-correction and few-shot learning.

Findings

Prompt self-correction with few-shot learning achieves highest F1 score of 0.79.

Topic-based and BERT models perform well on out-of-domain data.

Identified challenges in detecting subtle fairness issues in generated test content.

Abstract

Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Specifically, we review test content generated for a large-scale standardized English proficiency test with the goal of identifying content that only pertains to a certain subset of the test population as well as content that has the potential to be upsetting or distracting to some test takers. Issues like these could inadvertently impact a test taker's score and thus should be avoided. This kind of content does not reflect the more commonly-acknowledged biases, making it challenging even for modern models that contain safeguards. We build a dataset of 601 generated texts annotated for fairness and explore a variety of methods for classification: fine-tuning, topic-based classification, and prompting, including few-shot and self-correcting prompts. We find that combining prompt self-correction and few-shot learning performs best, yielding an F1 score of 0.79 on our held-out test set, while much smaller BERT- and topic-based models have competitive performance on out-of-domain data.