The Price of Diversity

TL;DR

This paper introduces a novel optimization method and an implementation tool to enhance diversity in decision-making processes while maintaining or improving meritocracy, demonstrated through real-world case studies.

Contribution

It proposes a new approach for balancing diversity and meritocracy by flipping outcome labels and using optimal classification trees for interpretability.

Findings

Diversity can be increased with minimal impact on meritocracy.

In some cases, diversity improvements also enhance meritocracy.

The method is effective across different real-world datasets.

Abstract

Systemic bias with respect to gender, race and ethnicity, often unconscious, is prevalent in datasets involving choices among individuals. Consequently, society has found it challenging to alleviate bias and achieve diversity in a way that maintains meritocracy in such settings. We propose (a) a novel optimization approach based on optimally flipping outcome labels and training classification models simultaneously to discover changes to be made in the selection process so as to achieve diversity without significantly affecting meritocracy, and (b) a novel implementation tool employing optimal classification trees to provide insights on which attributes of individuals lead to flipping of their labels, and to help make changes in the current selection processes in a manner understandable by human decision makers. We present case studies on three real-world datasets consisting of parole, admissions to the bar and lending decisions, and demonstrate that the price of diversity is low and sometimes negative, that is we can modify our selection processes in a way that enhances diversity without affecting meritocracy significantly, and sometimes improving it.