Discovering Differences in the Representation of People using Contextualized Semantic Axes

TL;DR

This paper extends the use of semantic axes to contextualized BERT embeddings to better identify social and temporal differences in representations of people, revealing evolving attitudes over time.

Contribution

It introduces a method for constructing contextualized semantic axes with BERT, addressing limitations of static embeddings in capturing nuanced social and temporal differences.

Findings

Contextualized axes distinguish differences in occupation representations.

References to women have become more negative over fourteen years.

The method effectively captures social and temporal semantic shifts.

Abstract

A common paradigm for identifying semantic differences across social and temporal contexts is the use of static word embeddings and their distances. In particular, past work has compared embeddings against "semantic axes" that represent two opposing concepts. We extend this paradigm to BERT embeddings, and construct contextualized axes that mitigate the pitfall where antonyms have neighboring representations. We validate and demonstrate these axes on two people-centric datasets: occupations from Wikipedia, and multi-platform discussions in extremist, men's communities over fourteen years. In both studies, contextualized semantic axes can characterize differences among instances of the same word type. In the latter study, we show that references to women and the contexts around them have become more detestable over time.