# Efficient Large-Scale Cross-Domain Sequential Recommendation with Dynamic State Representations

## TL;DR

This paper presents a scalable multi-domain recommendation approach that reduces computational costs by focusing attention within domains using novel positional embeddings and dynamic domain state representations, improving efficiency and performance.

## Contribution

It introduces Transition-Aware Positional Embeddings and Dynamic Domain State Representation to enhance multi-domain recommendation scalability and efficiency.

## Key findings

- Significant reduction in computational cost for multi-domain recommendation.
- Improved retrieval performance by modeling inter- and intra-domain information separately.
- Effective handling of large-scale, multi-domain recommendation challenges.

## Abstract

Recently, autoregressive recommendation models (ARMs), such as Meta's HSTU model, have emerged as a major breakthrough over traditional Deep Learning Recommendation Models (DLRMs), exhibiting the highly sought-after scaling law behaviour. However, when applied to multi-domain scenarios, the transformer architecture's attention maps become a computational bottleneck, as they attend to all items across every domain. To tackle this challenge, systems must efficiently balance inter and intra-domain knowledge transfer. In this work, we introduce a novel approach for scalable multi-domain recommendation systems by replacing full inter-domain attention with two innovative mechanisms: 1) Transition-Aware Positional Embeddings (TAPE): We propose novel positional embeddings that account for domain-transition specific information. This allows attention to be focused solely on intra-domain items, effectively reducing the unnecessary computational cost associated with attending to irrelevant domains. 2) Dynamic Domain State Representation (DDSR): We introduce a dynamic state representation for each domain, which is stored and accessed during subsequent token predictions. This enables the efficient transfer of relevant domain information without relying on full attention maps. Our method offers a scalable solution to the challenges posed by large-scale, multi-domain recommendation systems and demonstrates significant improvements in retrieval tasks by separately modelling and combining inter- and intra-domain representations.