Optimizing Encoder-Only Transformers for Session-Based Recommendation Systems

TL;DR

This paper introduces Sequential Masked Modeling, a novel encoder-only transformer approach that significantly improves session-based next-item recommendation by effectively capturing sequential dependencies through data augmentation and token masking.

Contribution

The paper presents a new transformer-based method, Sequential Masked Modeling, tailored for session-based recommendation, enhancing performance without requiring extensive user history.

Findings

Outperforms state-of-the-art models on three datasets

Effectively captures sequential dependencies in session data

Rivals methods with access to more user history

Abstract

Session-based recommendation is the task of predicting the next item a user will interact with, often without access to historical user data. In this work, we introduce Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation. Our method combines data augmentation through window sliding with a unique penultimate token masking strategy to capture sequential dependencies more effectively. By enhancing how transformers handle session data, Sequential Masked Modeling significantly improves next-item prediction performance. We evaluate our approach on three widely-used datasets, Yoochoose 1/64, Diginetica, and Tmall, comparing it to state-of-the-art single-session, cross-session, and multi-relation approaches. The results demonstrate that our Transformer-SMM models consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history. This study highlights the potential of encoder-only transformers in session-based recommendation and opens the door for further improvements.