VisualLens: Personalization through Task-Agnostic Visual History

TL;DR

VisualLens introduces a novel personalization framework leveraging users' daily life images and multimodal large language models, outperforming existing multimodal recommendation methods in accuracy and robustness.

Contribution

The paper presents VisualLens, a new approach that uses task-agnostic visual histories and multimodal models for personalized recommendations, along with two new benchmark datasets.

Findings

VisualLens improves recommendation accuracy by 5-10% on Hit@3.

It outperforms GPT-4o by 2-5%.

The method is robust across different history lengths and content categories.

Abstract

Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. However, item-based histories are not always accessible, and are not generalizable for multimodal recommendation. We hypothesize that a user's visual history -- comprising images from daily life -- can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization. To this end, we propose VisualLens, a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history. VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation. We created two new benchmarks, Google-Review-V and Yelp-V, with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10% on Hit@3, and outperforms GPT-4o by 2-5%. Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.