Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

TL;DR

This paper introduces a benchmark and evaluation measures for detecting social biases in sense embeddings, revealing biases at the sense level that are often overlooked by word-level assessments.

Contribution

It presents a novel benchmark dataset and bias evaluation measures specifically for sense embeddings, highlighting biases that are not captured by traditional word-level bias evaluations.

Findings

Sense embeddings exhibit social biases at the sense level.

Word-level bias measures may overlook sense-specific biases.

Biases are present even when word-level evaluations show no bias.

Abstract

Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures. We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures. Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense-level, which are often ignored by the word-level bias evaluation measures.