To See or To Read: User Behavior Reasoning in Multimodal LLMs

TL;DR

This paper introduces BehaviorLens, a benchmarking framework that compares textual and image representations of user behavior data in multimodal LLMs, revealing that image-based data significantly improves next-purchase prediction accuracy.

Contribution

The paper presents BehaviorLens, a systematic framework for evaluating modality trade-offs in user-behavior reasoning across multiple MLLMs using real-world purchase data.

Findings

Image representations improve prediction accuracy by 87.5%.

Text and image modalities have distinct advantages for user behavior reasoning.

Benchmarking reveals modality trade-offs in MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) are reshaping how modern agentic systems reason over sequential user-behavior data. However, whether textual or image representations of user behavior data are more effective for maximizing MLLM performance remains underexplored. We present \texttt{BehaviorLens}, a systematic benchmarking framework for assessing modality trade-offs in user-behavior reasoning across six MLLMs by representing transaction data as (1) a text paragraph, (2) a scatter plot, and (3) a flowchart. Using a real-world purchase-sequence dataset, we find that when data is represented as images, MLLMs next-purchase prediction accuracy is improved by 87.5% compared with an equivalent textual representation without any additional computational cost.