A Gray Literature Study on Fairness Requirements in AI-enabled Software Engineering

TL;DR

This study reviews gray literature on fairness in AI-enabled software engineering, highlighting diverse fairness definitions, management practices across the SDLC, and the societal impacts of fairness violations, emphasizing the need for standardized fairness frameworks.

Contribution

It provides a comprehensive analysis of how fairness requirements are defined, managed, and violated in AI software development through a gray literature review, highlighting gaps and implications.

Findings

Fairness definitions focus on non-discrimination and equal treatment.

Fairness management varies across SDLC stages, especially in bias mitigation.

Violations often stem from data bias, algorithm design, and transparency issues.

Abstract

Today, with the growing obsession with applying Artificial Intelligence (AI), particularly Machine Learning (ML), to software across various contexts, much of the focus has been on the effectiveness of AI models, often measured through common metrics such as F1- score, while fairness receives relatively little attention. This paper presents a review of existing gray literature, examining fairness requirements in AI context, with a focus on how they are defined across various application domains, managed throughout the Software Development Life Cycle (SDLC), and the causes, as well as the corresponding consequences of their violation by AI models. Our gray literature investigation shows various definitions of fairness requirements in AI systems, commonly emphasizing non-discrimination and equal treatment across different demographic and social attributes. Fairness requirement management practices vary across the SDLC, particularly in model training and bias mitigation, fairness monitoring and evaluation, and data handling practices. Fairness requirement violations are frequently linked, but not limited, to data representation bias, algorithmic and model design bias, human judgment, and evaluation and transparency gaps. The corresponding consequences include harm in a broad sense, encompassing specific professional and societal impacts as key examples, stereotype reinforcement, data and privacy risks, and loss of trust and legitimacy in AI-supported decisions. These findings emphasize the need for consistent frameworks and practices to integrate fairness into AI software, paying as much attention to fairness as to effectiveness.