Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework

TL;DR

This paper introduces SRP4CTR, a novel framework that leverages sequential recommendation pre-training to improve CTR prediction accuracy while managing inference costs effectively.

Contribution

The paper proposes a new pre-training and fine-tuning approach with cross-attention and querying transformer techniques for better knowledge transfer in CTR tasks.

Findings

Outperforms baseline models in offline experiments

Effective transfer of pre-trained knowledge to CTR models

Maintains low inference costs during deployment

Abstract

Understanding user interests is crucial for Click-Through Rate (CTR) prediction tasks. In sequential recommendation, pre-training from user historical behaviors through self-supervised learning can better comprehend user dynamic preferences, presenting the potential for direct integration with CTR tasks. Previous methods have integrated pre-trained models into downstream tasks with the sole purpose of extracting semantic information or well-represented user features, which are then incorporated as new features. However, these approaches tend to ignore the additional inference costs to the downstream tasks, and they do not consider how to transfer the effective information from the pre-trained models for specific estimated items in CTR prediction. In this paper, we propose a Sequential Recommendation Pre-training framework for CTR prediction (SRP4CTR) to tackle the above problems. Initially, we discuss the impact of introducing pre-trained models on inference costs. Subsequently, we introduced a pre-trained method to encode sequence side information concurrently.During the fine-tuning process, we incorporate a cross-attention block to establish a bridge between estimated items and the pre-trained model at a low cost. Moreover, we develop a querying transformer technique to facilitate the knowledge transfer from the pre-trained model to industrial CTR models. Offline and online experiments show that our method outperforms previous baseline models.