Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers

TL;DR

This paper identifies the limitations of inverse propensity scoring in sequential recommendation with bidirectional Transformers and proposes a new propensity scoring method to effectively debias the Cloze task, improving recommendation accuracy.

Contribution

It introduces a novel propensity scoring mechanism specifically designed for sequential recommendation, addressing exposure bias in bidirectional Transformer models.

Findings

The proposed method effectively debiases the Cloze task in experiments.

It is robust against varying levels of exposure bias.

Outperforms traditional IPS in sequential recommendation scenarios.

Abstract

Bidirectional Transformer architectures are state-of-the-art sequential recommendation models that use a bi-directional representation capacity based on the Cloze task, a.k.a. Masked Language Modeling. The latter aims to predict randomly masked items within the sequence. Because they assume that the true interacted item is the most relevant one, an exposure bias results, where non-interacted items with low exposure propensities are assumed to be irrelevant. The most common approach to mitigating exposure bias in recommendation has been Inverse Propensity Scoring (IPS), which consists of down-weighting the interacted predictions in the loss function in proportion to their propensities of exposure, yielding a theoretically unbiased learning. In this work, we argue and prove that IPS does not extend to sequential recommendation because it fails to account for the temporal nature of the problem. We then propose a novel propensity scoring mechanism, which can theoretically debias the Cloze task in sequential recommendation. Finally we empirically demonstrate the debiasing capabilities of our proposed approach and its robustness to the severity of exposure bias.