Catching Dark Signals in Algorithms: Unveiling Audiovisual and Thematic Markers of Unsafe Content Recommended for Children and Teenagers

TL;DR

This study analyzes recommended short videos for children and teenagers across major platforms, revealing that unsafe content often contains darker visuals and implicit harm, emphasizing the need for better moderation and regulation.

Contribution

It introduces a multimodal analysis framework and thematic modeling to identify explicit and implicit online harms in short videos recommended to youth.

Findings

Unsafe videos have darker visual features

Explicit harmful content is prevalent in recommended videos

Implicit harm from ordinary content induces anxiety

Abstract

The prevalence of short form video platforms, combined with the ineffectiveness of age verification mechanisms, raises concerns about the potential harms facing children and teenagers in an algorithm-moderated online environment. We conducted multimodal feature analysis and thematic topic modeling of 4,492 short videos recommended to children and teenagers on Instagram Reels, TikTok, and YouTube Shorts, collected as a part of an algorithm auditing experiment. This feature-level and content-level analysis revealed that unsafe (i.e., problematic, mentally distressing) short videos (a) possess darker visual features and (b) contain explicitly harmful content and implicit harm from anxiety-inducing ordinary content. We introduce a useful framework of online harm (i.e., explicit, implicit, unintended), providing a unique lens for understanding the dynamic, multifaceted online risks facing children and teenagers. The findings highlight the importance of protecting younger audiences in critical developmental stages from both explicit and implicit risks on social media, calling for nuanced content moderation, age verification, and platform regulation.