A Capabilities Approach to Studying Bias and Harm in Language Technologies

TL;DR

This paper advocates for using the Capabilities Approach to evaluate bias and harm in Language Technologies, emphasizing community needs and intersectional contexts over resource-based metrics.

Contribution

It introduces the Capabilities Approach as a framework for assessing fairness and harm in multilingual NLP, promoting community-centered evaluation methods.

Findings

Highlights limitations of resource-based fairness metrics

Proposes a community-centered evaluation framework

Connects the Capabilities Approach to multilingual NLP assessment

Abstract

Mainstream Natural Language Processing (NLP) research has ignored the majority of the world's languages. In moving from excluding the majority of the world's languages to blindly adopting what we make for English, we first risk importing the same harms we have at best mitigated and at least measured for English. However, in evaluating and mitigating harms arising from adopting new technologies into such contexts, we often disregard (1) the actual community needs of Language Technologies, and (2) biases and fairness issues within the context of the communities. In this extended abstract, we consider fairness, bias, and inclusion in Language Technologies through the lens of the Capabilities Approach. The Capabilities Approach centers on what people are capable of achieving, given their intersectional social, political, and economic contexts instead of what resources are (theoretically) available to them. We detail the Capabilities Approach, its relationship to multilingual and multicultural evaluation, and how the framework affords meaningful collaboration with community members in defining and measuring the harms of Language Technologies.