Break the Inaccessible Boundary: Distilling Post-Conversion Content for User Retention Modeling

TL;DR

This paper introduces OCARM, a two-stage distillation framework that enhances user retention prediction by implicitly capturing future onboarding content signals without feature leakage.

Contribution

The paper proposes a novel distillation-based method to incorporate future onboarding content into retention models while avoiding feature leakage during training.

Findings

Improved retention prediction accuracy in offline experiments.

Consistent online A/B test improvements in real-world scenarios.

Abstract

User retention is a key metric to measure long-term engagement in modern platforms. In real-time bidding (RTB) advertising system for user re-engagement, the retention model is required to predict future revisit probability at bidding time, before the user converts and consumes any content. Although post-conversion content, termed Onboarding Content, provides highly informative signals for retention prediction, directly using it in training causes severe feature leakage and creates a gap between training and serving. To address this issue, we propose OCARM, a two-stage distillation-aligned framework for Onboarding Content Augmented Retention Modeling, enabling the model to implicitly capture future content using only observable features during inference. In the first stage, we deliberately expose onboarding content to train a hierarchical encoder that produces teacher representations. In the second stage, a user encoder is aligned with the frozen teacher through distillation, allowing the model to approximate the inaccessible onboarding signals without leakage. Extensive offline experiments and online A/B tests demonstrate that our framework achieves consistent improvements in a real-world growth scenario.