HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

TL;DR

HORIZON is a comprehensive benchmark for user behavior modeling that emphasizes cross-domain, long-term, and generalizable scenarios, addressing limitations of existing narrow benchmarks.

Contribution

It introduces a large-scale, cross-domain dataset and new evaluation tasks that better reflect real-world user modeling challenges.

Findings

Current models struggle with cross-domain and long-term generalization.

Benchmarking reveals gaps between existing methods and real-world demands.

HORIZON provides a foundation for developing more robust user models.

Abstract

User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focusing mainly on short sessions and next-item prediction within a single domain. Such limitations hinder progress toward robust and generalizable user models. We present HORIZON, a new benchmark that reformulates user modeling along three axes i.e. dataset, task, and evaluation. Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items, enabling both pretraining and realistic evaluation of models in heterogeneous environments. Unlike prior benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain. We propose new tasks and evaluation setups that better reflect real-world deployment scenarios. These include temporal generalization, sequence-length variation, and modeling unseen users, with metrics designed to assess general user behavior understanding rather than isolated next-item prediction. We benchmark popular sequential recommendation architectures alongside LLM-based baselines that leverage long-term interaction histories. Our results highlight the gap between current methods and the demands of real-world user modeling, while establishing HORIZON as a foundation for research on temporally robust, cross-domain, and general-purpose user models.