# Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark

## TL;DR

This paper introduces RecBench-MD, a comprehensive benchmark for evaluating the recommendation capabilities of foundation models across multiple datasets and domains, highlighting the importance of fine-tuning and multi-domain training.

## Contribution

The study presents RecBench-MD, a new benchmark for assessing foundation models' recommendation abilities across diverse datasets and domains, with extensive evaluations of 19 models.

## Key findings

- In-domain fine-tuning yields the best performance.
- Cross-dataset transfer learning supports new recommendation scenarios.
- Multi-domain training improves model adaptability.

## Abstract

Comprehensive evaluation of the recommendation capabilities of existing foundation models across diverse datasets and domains is essential for advancing the development of recommendation foundation models. In this study, we introduce RecBench-MD, a novel and comprehensive benchmark designed to assess the recommendation abilities of foundation models from a zero-resource, multi-dataset, and multi-domain perspective. Through extensive evaluations of 19 foundation models across 15 datasets spanning 10 diverse domains -- including e-commerce, entertainment, and social media -- we identify key characteristics of these models in recommendation tasks. Our findings suggest that in-domain fine-tuning achieves optimal performance, while cross-dataset transfer learning provides effective practical support for new recommendation scenarios. Additionally, we observe that multi-domain training significantly enhances the adaptability of foundation models. All code and data have been publicly released to facilitate future research.