UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation

TL;DR

UniTRec introduces a unified Transformer framework that models user history at local and global levels and uses contrastive learning with perplexity estimation to improve text-based recommendation performance.

Contribution

It proposes a novel unified Transformer-based model that combines local and global attention with contrastive learning for enhanced recommendation accuracy.

Findings

Achieves state-of-the-art results on three text-based recommendation tasks.

Effectively models user history with a dual-level attention mechanism.

Utilizes contrastive signals from language perplexity to improve user-item matching.

Abstract

Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation. In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multi-turn history representations, we propose a unified local- and global-attention Transformer encoder to better model two-level contexts of user history. Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching. Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks. Code is available at https://github.com/Veason-silverbullet/UniTRec.