Perception-Driven Bias Detection in Machine Learning via Crowdsourced Visual Judgment

TL;DR

This paper proposes a perception-driven, crowdsourced framework for bias detection in machine learning, using visual judgments from non-experts to identify disparities without relying on sensitive labels.

Contribution

It introduces a novel, scalable approach leveraging visual perception and crowdsourcing to detect biases, reducing dependence on labeled data and complex fairness metrics.

Findings

Crowdsourced visual judgments reliably signal known biases.

Perceptual cues correlate with statistical bias measures.

The method offers an interpretable, label-efficient bias detection alternative.

Abstract

Machine learning systems are increasingly deployed in high-stakes domains, yet they remain vulnerable to bias systematic disparities that disproportionately impact specific demographic groups. Traditional bias detection methods often depend on access to sensitive labels or rely on rigid fairness metrics, limiting their applicability in real-world settings. This paper introduces a novel, perception-driven framework for bias detection that leverages crowdsourced human judgment. Inspired by reCAPTCHA and other crowd-powered systems, we present a lightweight web platform that displays stripped-down visualizations of numeric data (for example-salary distributions across demographic clusters) and collects binary judgments on group similarity. We explore how users' visual perception-shaped by layout, spacing, and question phrasing can signal potential disparities. User feedback is aggregated to flag data segments as biased, which are then validated through statistical tests and machine learning cross-evaluations. Our findings show that perceptual signals from non-expert users reliably correlate with known bias cases, suggesting that visual intuition can serve as a powerful, scalable proxy for fairness auditing. This approach offers a label-efficient, interpretable alternative to conventional fairness diagnostics, paving the way toward human-aligned, crowdsourced bias detection pipelines.