Beyond Binary Classification: Detecting Fine-Grained Sexism in Social Media Videos

TL;DR

This paper introduces FineMuSe, a multimodal dataset with fine-grained labels for sexism detection in Spanish social media videos, and evaluates LLMs' ability to identify nuanced sexism forms.

Contribution

It presents a new dataset, a hierarchical taxonomy for sexism, and an evaluation of LLMs on fine-grained sexism detection tasks.

Findings

Multimodal LLMs perform comparably to humans in detecting nuanced sexism.

LLMs struggle with co-occurring sexist types conveyed visually.

The dataset enables more detailed sexism analysis in social media videos.

Abstract

Online sexism appears in various forms, which makes its detection challenging. Although automated tools can enhance the identification of sexist content, they are often restricted to binary classification. Consequently, more subtle manifestations of sexism may remain undetected due to the lack of fine-grained, context-sensitive labels. To address this issue, we make the following contributions: (1) we present FineMuSe, a new multimodal sexism detection dataset in Spanish that includes both binary and fine-grained annotations; (2) we introduce a comprehensive hierarchical taxonomy that encompasses forms of sexism, non-sexism, and rhetorical devices of irony and humor; and (3) we evaluate a wide range of LLMs for both binary and fine-grained sexism detection. Our findings indicate that multimodal LLMs perform competitively with human annotators in identifying nuanced forms of sexism; however, they struggle to capture co-occurring sexist types when these are conveyed through visual cues.