CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction

TL;DR

CTR-Sink introduces behavior-level attention sinks in language models to improve click-through rate prediction by better capturing user behavior boundaries and relationships, addressing the semantic fragmentation issue.

Contribution

It proposes a novel attention sink framework with behavior-level sinks and a two-stage training strategy tailored for recommendation tasks.

Findings

Improves CTR prediction accuracy on industrial and open datasets.

Effectively captures behavioral correlations through attention sinks.

Visualizations demonstrate better attention focus on meaningful behavior boundaries.

Abstract

Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios.