Comparative Separation: Evaluating Separation on Comparative Judgment Test Data
Xiaoyin Xi, Neeku Capak, Kate Stockwell, Zhe Yu

TL;DR
This paper introduces comparative separation, a new fairness evaluation method using comparative judgment test data, which reduces human labeling effort and is shown to be equivalent to traditional separation in binary classification.
Contribution
It defines the novel fairness notion of comparative separation, develops evaluation metrics, and demonstrates its theoretical and empirical equivalence to separation in binary classification.
Findings
Comparative separation is equivalent to separation in binary classification.
Using comparative judgment data reduces labeling effort.
The method provides a practical way to evaluate fairness without ground truth labels.
Abstract
This research seeks to benefit the software engineering society by proposing comparative separation, a novel group fairness notion to evaluate the fairness of machine learning software on comparative judgment test data. Fairness issues have attracted increasing attention since machine learning software is increasingly used for high-stakes and high-risk decisions. It is the responsibility of all software developers to make their software accountable by ensuring that the machine learning software do not perform differently on different sensitive groups -- satisfying the separation criterion. However, evaluation of separation requires ground truth labels for each test data point. This motivates our work on analyzing whether separation can be evaluated on comparative judgment test data. Instead of asking humans to provide the ratings or categorical labels on each test data point,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEthics and Social Impacts of AI · Explainable Artificial Intelligence (XAI) · Mobile Crowdsensing and Crowdsourcing
