From Perceived Effectiveness to Measured Impact: Identity-Aware Evaluation of Automated Counter-Stereotypes

TL;DR

This study evaluates the real-world impact of automated counter-stereotypes on gender bias across demographics, revealing complex, identity-sensitive effects and the divergence between perceived and actual effectiveness.

Contribution

It introduces an identity-aware evaluation framework for automated counter-stereotypes, highlighting nuanced demographic differences in bias mitigation effectiveness.

Findings

Older, male participants showed measurable implicit bias reduction.

Younger women exhibited increased bias after interventions.

Perceived effectiveness often did not match actual bias reduction results.

Abstract

We investigate the effect of automatically generated counter-stereotypes on gender bias held by users of various demographics on social media. Building on recent NLP advancements and social psychology literature, we evaluate two counter-stereotype strategies -- counter-facts and broadening universals (i.e., stating that anyone can have a trait regardless of group membership) -- which have been identified as the most potentially effective in previous studies. We assess the real-world impact of these strategies on mitigating gender bias across user demographics (gender and age), through the Implicit Association Test and the self-reported measures of explicit bias and perceived utility. Our findings reveal that actual effectiveness does not align with perceived effectiveness, and the former is a nuanced and sometimes divergent phenomenon across demographic groups. While overall bias reduction was limited, certain groups (e.g., older, male participants) exhibited measurable improvements in implicit bias in response to some interventions. Conversely, younger participants, especially women, showed increasing bias in response to the same interventions. These results highlight the complex and identity-sensitive nature of stereotype mitigation and call for dynamic and context-aware evaluation and mitigation strategies.