Detecting Sexism in German Online Newspaper Comments with Open-Source Text Embeddings (Team GDA, GermEval2024 Shared Task 1: GerMS-Detect, Subtasks 1 and 2, Closed Track)

TL;DR

This paper presents a method using open-source text embeddings to detect sexism in German online comments, achieving competitive results in a shared task and demonstrating potential for scalable, multilingual applications.

Contribution

It introduces a classifier trained on text embeddings that closely mimics human judgments of sexism, showing robust performance in a competitive benchmark.

Findings

Achieved an average macro F1 score of 0.597 in sexism detection

Predicted human annotation distributions with Jensen-Shannon distance of 0.301

Demonstrated computational efficiency suitable for scalable multilingual use

Abstract

Sexism in online media comments is a pervasive challenge that often manifests subtly, complicating moderation efforts as interpretations of what constitutes sexism can vary among individuals. We study monolingual and multilingual open-source text embeddings to reliably detect sexism and misogyny in German-language online comments from an Austrian newspaper. We observed classifiers trained on text embeddings to mimic closely the individual judgements of human annotators. Our method showed robust performance in the GermEval 2024 GerMS-Detect Subtask 1 challenge, achieving an average macro F1 score of 0.597 (4th place, as reported on Codabench). It also accurately predicted the distribution of human annotations in GerMS-Detect Subtask 2, with an average Jensen-Shannon distance of 0.301 (2nd place). The computational efficiency of our approach suggests potential for scalable applications across various languages and linguistic contexts.