Markovian Pre-Trained Transformer for Next-Item Recommendation

TL;DR

The paper proposes a novel pre-trained transformer model, MPT, trained on synthetic Markov chains, which effectively captures sequential user behavior for next-item recommendation, outperforming traditional methods.

Contribution

Introducing MPT, a universal, transferable pre-trained transformer for recommendations, trained on synthetic Markov chains to improve sequence summarization and prediction accuracy.

Findings

MPT achieves state-of-the-art performance on five datasets.

Pre-training on synthetic Markov chains enhances recommendation accuracy.

MPT outperforms traditional and language pre-training methods.

Abstract

We introduce the Markovian Pre-trained Transformer (MPT) for next-item recommendation, a transferable model fully pre-trained on synthetic Markov chains, yet capable of achieving state-of-the-art performance by fine-tuning a lightweight adaptor. This counterintuitive success stems from the observation of the `Markovian' nature: advanced sequential recommenders coincidentally rely on the latest interaction to make predictions, while the historical interactions serve mainly as auxiliary cues for inferring the user's general, non-sequential identity. This characteristic necessitates the capabilities of a universal recommendation model to effectively summarize the user sequence, with particular emphasis on the latest interaction. MPT inherently has the potential to be universal and transferable. On the one hand, when trained to predict the next state of Markov chains, it acquires the capabilities to estimate transition probabilities from the context (one adaptive manner for summarizing sequences) and attend to the last state to ensure accurate state transitions. On the other hand, unlike the heterogeneous interaction data, an unlimited amount of controllable Markov chains is available to boost the model capacity. We conduct extensive experiments on five public datasets from three distinct platforms to validate the superiority of Markovian pre-training over traditional recommendation pre-training and recent language pre-training paradigms.