Auditing Algorithmic Personalization in TikTok Comment Sections

TL;DR

This study investigates how TikTok's comment sections are personalized based on political preferences, revealing that personalization can influence comment exposure and varies with video engagement and content.

Contribution

It provides the first empirical analysis of political comment personalization on TikTok, demonstrating its dependence on video engagement and content context.

Findings

Comment ranking divergence varies with political group and video engagement.

Personalization can lead to politically aligned comment exposure.

The extent of personalization is context-dependent.

Abstract

Personalization algorithms are ubiquitous in modern social computing systems, yet their effects on comment sections remain underexplored. In this work, we conducted an algorithmic auditing experiment to examine comment personalization on TikTok. We trained sock-puppet accounts to exhibit left-leaning or right-leaning preferences and successfully validated 17 of them by analyzing the videos recommended on their For You Pages. We then scraped the comment sections shown to these trained partisan accounts, along with five cold-start accounts, across 65 politically neutral videos related to the 2024 U.S. presidential election that contain abundant discussions from both left-leaning and right-leaning perspectives. We find that while the composition of top comments remains largely consistent for all videos, ranking divergence between accounts from different political groups is significantly greater than that observed within the same group for some videos. This effect is strongly correlated with video-level metrics such as comment volume, engagement inequality, and partisan skew in the comment sections. Furthermore, through an exploratory case study, we find preliminary evidence that personalization can result in comment exposure aligned with an account's political leaning. However, this pattern is not universal, suggesting that the extent of politically oriented comment personalization is context-dependent.