Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction

TL;DR

CrossAdapt is a two-stage framework that enables efficient transfer of knowledge across different model architectures in large-scale user response prediction, reducing retraining costs and improving performance.

Contribution

It introduces a novel two-stage approach with dimension-adaptive projections and asymmetric co-distillation for effective cross-architecture knowledge transfer.

Findings

Achieves 0.27-0.43% AUC improvements on public datasets.

Reduces training time by 43-71%.

Mitigates AUC degradation and bias in large-scale deployment.

Abstract

Deploying new architectures in large-scale user response prediction systems incurs high model switching costs due to expensive retraining on massive historical data and performance degradation under data retention constraints. Existing knowledge distillation methods struggle with architectural heterogeneity and the prohibitive cost of transferring large embedding tables. We propose CrossAdapt, a two-stage framework for efficient cross-architecture knowledge transfer. The offline stage enables rapid embedding transfer via dimension-adaptive projections without iterative training, combined with progressive network distillation and strategic sampling to reduce computational cost. The online stage introduces asymmetric co-distillation, where students update frequently while teachers update infrequently, together with a distribution-aware adaptation mechanism that dynamically balances historical knowledge preservation and fast adaptation to evolving data. Experiments on three public datasets show that CrossAdapt achieves 0.27-0.43% AUC improvements while reducing training time by 43-71%. Large-scale deployment on Tencent WeChat Channels (~10M daily samples) further demonstrates its effectiveness, significantly mitigating AUC degradation, LogLoss increase, and prediction bias compared to standard distillation baselines.