Rotate Both Ways: Time-and-Order RoPE for Generative Recommendation

TL;DR

This paper introduces Time-and-Order RoPE (TO-RoPE), a novel rotary position embedding method that effectively models both temporal and sequential information in generative recommendation systems, outperforming existing approaches.

Contribution

The paper proposes TO-RoPE, a new family of rotary position embeddings that jointly encode event time and sequence index, enhancing generative recommendation accuracy.

Findings

TO-RoPE variants outperform existing methods in accuracy.

Extensive experiments validate the effectiveness of TO-RoPE on multiple datasets.

Rotary embeddings provide a simple and deployment-friendly foundation for generative recommendation.

Abstract

Generative recommenders, typically transformer-based autoregressive models, predict the next item or action from a user's interaction history. Their effectiveness depends on how the model represents where an interaction event occurs in the sequence (discrete index) and when it occurred in wall-clock time. Prevailing approaches inject time via learned embeddings or relative attention biases. In this paper, we argue that RoPE-based approaches, if designed properly, can be a stronger alternative for jointly modeling temporal and sequential information in user behavior sequences. While vanilla RoPE in LLMs considers only token order, generative recommendation requires incorporating both event time and token index. To address this, we propose Time-and-Order RoPE (TO-RoPE), a family of rotary position embedding designs that treat index and time as angle sources shaping the query-key geometry directly. We present three instantiations: early fusion, split-by-dim, and split-by-head. Extensive experiments on both publicly available datasets and a proprietary industrial dataset show that TO-RoPE variants consistently improve accuracy over existing methods for encoding time and index. These results position rotary embeddings as a simple, principled, and deployment-friendly foundation for generative recommendation.