Sexism in the Judiciary

TL;DR

This study analyzes 6.7 million legal documents to detect gender bias, highlights limitations of existing NLP methods, and introduces automated approaches that outperform current techniques, revealing gender biases in influential case law.

Contribution

It proposes two new automated methods for creating bias-related word lists, improving NLP bias detection accuracy in legal texts.

Findings

Existing NLP bias detection methods are inconsistent and rely on subjective word lists.

The proposed methods outperform current NLP bias detection algorithms.

Gender bias in case law correlates with historical women's workforce participation.

Abstract

We analyze 6.7 million case law documents to determine the presence of gender bias within our judicial system. We find that current bias detectino methods in NLP are insufficient to determine gender bias in our case law database and propose an alternative approach. We show that existing algorithms' inconsistent results are consequences of prior research's definition of biases themselves. Bias detection algorithms rely on groups of words to represent bias (e.g., 'salary,' 'job,' and 'boss' to represent employment as a potentially biased theme against women in text). However, the methods to build these groups of words have several weaknesses, primarily that the word lists are based on the researchers' own intuitions. We suggest two new methods of automating the creation of word lists to represent biases. We find that our methods outperform current NLP bias detection methods. Our research improves the capabilities of NLP technology to detect bias and highlights gender biases present in influential case law. In order test our NLP bias detection method's performance, we regress our results of bias in case law against U.S census data of women's participation in the workforce in the last 100 years.