The Value of AI-Generated Metadata for UGC Platforms: Evidence from a Large-scale Field Experiment

TL;DR

This large-scale experiment demonstrates that AI-generated titles significantly increase content consumption and improve user-video matching on UGC platforms, especially when combined with human editing.

Contribution

The study provides empirical evidence of the positive impact of AI-generated metadata on user engagement and introduces insights into human-AI co-creation benefits.

Findings

AI-generated titles increased valid watches by 1.6% and watch duration by 0.9%.

Adoption of AI-generated titles by producers led to 7.1% more views.

Human-AI co-created titles outperformed fully AI or human-generated titles.

Abstract

AI-generated content (AIGC), such as advertisement copy, product descriptions, and social media posts, is becoming ubiquitous in business practices. However, the value of AI-generated metadata, such as titles, remains unclear on user-generated content (UGC) platforms. To address this gap, we conducted a large-scale field experiment on a leading short-video platform in Asia to provide about 1 million users access to AI-generated titles for their uploaded videos. Our findings show that the provision of AI-generated titles significantly boosted content consumption, increasing valid watches by 1.6% and watch duration by 0.9%. When producers adopted these titles, these increases jumped to 7.1% and 4.1%, respectively. This viewership-boost effect was largely attributed to the use of this generative AI (GAI) tool increasing the likelihood of videos having a title by 41.4%. The effect was more pronounced for groups more affected by metadata sparsity. Mechanism analysis revealed that AI-generated metadata improved user-video matching accuracy in the platform's recommender system. Interestingly, for a video for which the producer would have posted a title anyway, adopting the AI-generated title decreased its viewership on average, implying that AI-generated titles may be of lower quality than human-generated ones. However, when producers chose to co-create with GAI and significantly revised the AI-generated titles, the videos outperformed their counterparts with either fully AI-generated or human-generated titles, showcasing the benefits of human-AI co-creation. This study highlights the value of AI-generated metadata and human-AI metadata co-creation in enhancing user-content matching and content consumption for UGC platforms.