SCOPE: A Dataset of Stereotyped Prompts for Counterfactual Fairness Assessment of LLMs

TL;DR

SCOPE is a large, diverse dataset of counterfactual prompts designed to systematically evaluate fairness and bias in large language models across multiple topics, bias dimensions, and communicative intents.

Contribution

The paper introduces SCOPE, a comprehensive dataset of 241,280 prompts with 120,640 counterfactual pairs, enabling detailed analysis of demographic bias and fairness in LLMs.

Findings

SCOPE covers 1,438 topics and 1,536 demographic groups.

Prompts are generated under four communicative intents for broad interaction coverage.

The dataset facilitates systematic evaluation of bias, robustness, and counterfactual consistency in LLMs.

Abstract

Large Language Models (LLMs) now serve as the foundation for a wide range of applications, from conversational assistants to decision support tools, making the issue of fairness in their results increasingly important. Previous studies have shown that LLM outputs can shift when prompts reference different demographic groups, even when intent and semantic content remain constant. However, existing resources for probing such disparities rely primarily on small, template-based counterfactual examples or fixed sentence pairs. These benchmarks offer limited linguistic diversity, narrow topical coverage, and little support for analyzing how communicative intent affects model behavior. To address these limitations, we introduce SCOPE (Stereotype-COnditioned Prompts for Evaluation), a large-scale dataset of counterfactual prompt pairs designed to enable systematic investigation of group-sensitive behavior in LLMs. SCOPE contains 241,280 prompts organized into 120,640 counterfactual pairs, each grounded in one of 1,438 topics and spanning nine bias dimensions and 1,536 demographic groups. All prompts are generated under four distinct communicative intents: Question, Recommendation, Direction, and Clarification, ensuring broad coverage of common interaction styles. This resource provides a controlled, semantically aligned, and intent-aware basis for evaluating fairness, robustness, and counterfactual consistency.