Conditional Quantile Estimation for Uncertain Watch Time in Short-Video Recommendation

TL;DR

This paper introduces Conditional Quantile Estimation (CQE), a novel method for modeling the entire distribution of user watch time in short-video platforms, leading to improved prediction accuracy and user engagement.

Contribution

The paper presents CQE, a new quantile regression-based approach that captures the full watch-time distribution, outperforming existing methods in both offline and online evaluations.

Findings

CQE improves watch-time prediction accuracy.

CQE enhances user engagement metrics in online deployment.

CQE outperforms traditional average-based models.

Abstract

Accurately predicting watch time is crucial for optimizing recommendations and user experience in short video platforms. However, existing methods that estimate a single average watch time often fail to capture the inherent uncertainty in user engagement patterns. In this paper, we propose Conditional Quantile Estimation (CQE) to model the entire conditional distribution of watch time. Using quantile regression, CQE characterizes the complex watch-time distribution for each user-video pair, providing a flexible and comprehensive approach to understanding user behavior. We further design multiple strategies to combine the quantile estimates, adapting to different recommendation scenarios and user preferences. Extensive offline experiments and online A/B tests demonstrate the superiority of CQE in watch-time prediction and user engagement modeling. Specifically, deploying CQE online on a large-scale platform with hundreds of millions of daily active users has led to substantial gains in key evaluation metrics, including active days, engagement time, and video views. These results highlight the practical impact of our proposed approach in enhancing the user experience and overall performance of the short video recommendation system. The code will be released https://github.com/justopit/CQE.