Not Judging a User by Their Cover: Understanding Harm in Multi-Modal Processing within Social Media Research

TL;DR

This paper highlights the importance of multi-modal processing in social media research, revealing biases and ethical concerns in content moderation tools and human annotation practices across different media types.

Contribution

It emphasizes the need to consider multimedia modalities and ethical implications in social media harm detection, supported by empirical analysis of annotator performance and bias.

Findings

Annotator performance varies significantly across modalities.

Biases in demographic prediction influence harm detection.

Auditing Perspective API reveals demographic disparities.

Abstract

Social media has shaken the foundations of our society, unlikely as it may seem. Many of the popular tools used to moderate harmful digital content, however, have received widespread criticism from both the academic community and the public sphere for middling performance and lack of accountability. Though social media research is thought to center primarily on natural language processing, we demonstrate the need for the community to understand multimedia processing and its unique ethical considerations. Specifically, we identify statistical differences in the performance of Amazon Turk (MTurk) annotators when different modalities of information are provided and discuss the patterns of harm that arise from crowd-sourced human demographic prediction. Finally, we discuss the consequences of those biases through auditing the performance of a toxicity detector called Perspective API on the language of Twitter users across a variety of demographic categories.