Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation

TL;DR

This paper identifies and addresses the representation degeneration problem in sequential recommendation models by proposing DuoRec, a contrastive learning-based approach that improves embedding distribution and recommendation performance.

Contribution

The paper introduces DuoRec, a novel contrastive regularization method with model-level augmentation and a new sampling strategy to enhance sequential recommendation models.

Findings

DuoRec outperforms baseline methods on five datasets.

Representation degeneration is significantly alleviated by DuoRec.

Visualization confirms improved embedding distribution.

Abstract

Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.