Generative Representational Learning of Foundation Models for Recommendation

TL;DR

RecFound introduces a comprehensive framework for recommendation foundation models, addressing multi-task learning challenges with novel training schemes, and achieves state-of-the-art results across diverse recommendation tasks.

Contribution

It presents the first dataset and a multi-task training scheme with innovative modules to improve recommendation foundation models' performance.

Findings

RecFound outperforms existing baselines on multiple recommendation tasks.

The proposed modules effectively handle knowledge sharing and conflict.

RecFound demonstrates improved convergence speed and task balance.

Abstract

Developing a single foundation model with the capability to excel across diverse tasks has been a long-standing objective in the field of artificial intelligence. As the wave of general-purpose foundation models sweeps across various domains, their influence has significantly extended to the field of recommendation systems. While recent efforts have explored recommendation foundation models for various generative tasks, they often overlook crucial embedding tasks and struggle with the complexities of multi-task learning, including knowledge sharing & conflict resolution, and convergence speed inconsistencies. To address these limitations, we introduce RecFound, a generative representational learning framework for recommendation foundation models. We construct the first comprehensive dataset for recommendation foundation models covering both generative and embedding tasks across diverse scenarios. Based on this dataset, we propose a novel multi-task training scheme featuring a Task-wise Mixture of Low-rank Experts (TMoLE) to handle knowledge sharing & conflict, a Step-wise Convergence-oriented Sample Scheduler (S2Sched) to address inconsistent convergence, and a Model Merge module to balance the performance across tasks. Experiments demonstrate that RecFound achieves state-of-the-art performance across various recommendation tasks, outperforming existing baselines.