Evaluating AI Group Fairness: a Fuzzy Logic Perspective

TL;DR

This paper introduces a fuzzy logic framework to evaluate and interpret group fairness definitions in AI, accommodating social context and uncertainty, and providing a continuous measure of fairness rather than binary judgments.

Contribution

It develops a novel fuzzy logic-based approach to formalize and analyze AI fairness definitions, allowing for context-sensitive and uncertain evaluations.

Findings

Fuzzy logic enables continuous evaluation of fairness measures.

The framework rationalizes and reinterprets existing fairness definitions.

It facilitates layperson understanding of complex fairness concepts.

Abstract

Artificial intelligence systems often address fairness concerns by evaluating and mitigating measures of group discrimination, for example that indicate biases against certain genders or races. However, what constitutes group fairness depends on who is asked and the social context, whereas definitions are often relaxed to accept small deviations from the statistical constraints they set out to impose. Here we decouple definitions of group fairness both from the context and from relaxation-related uncertainty by expressing them in the axiomatic system of Basic fuzzy Logic (BL) with loosely understood predicates, like encountering group members. We then evaluate the definitions in subclasses of BL, such as Product or Lukasiewicz logics. Evaluation produces continuous instead of binary truth values by choosing the logic subclass and truth values for predicates that reflect uncertain context-specific beliefs, such as stakeholder opinions gathered through questionnaires. Internally, it follows logic-specific rules to compute the truth values of definitions. We show that commonly held propositions standardize the resulting mathematical formulas and we transcribe logic and truth value choices to layperson terms, so that anyone can answer them. We also use our framework to study several literature definitions of algorithmic fairness, for which we rationalize previous expedient practices that are non-probabilistic and show how to re-interpret their formulas and parameters in new contexts.