VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

TL;DR

VehicleMemBench is a new benchmark designed to evaluate multi-user long-term memory and decision-making in in-vehicle agents, addressing the limitations of existing static, single-user benchmarks.

Contribution

It introduces a comprehensive, executable simulation benchmark with multi-user memory modeling, tool interaction, and dynamic preference evolution, enabling more realistic evaluation of in-vehicle AI systems.

Findings

Powerful models excel at direct instructions but struggle with memory evolution.

Advanced memory systems face challenges in domain-specific, long-term memory tasks.

Dynamic user preferences significantly impact model performance.

Abstract

With the growing demand for intelligent in-vehicle experiences, vehicle-based agents are evolving from simple assistants to long-term companions. This evolution requires agents to continuously model multi-user preferences and make reliable decisions in the face of inter-user preference conflicts and changing habits over time. However, existing benchmarks are largely limited to single-user, static question-answer settings, failing to capture the temporal evolution of preferences and the multi-user, tool-interactive nature of real vehicle environments. To address this gap, we introduce VehicleMemBench, a multi-user long-context memory benchmark built on an executable in-vehicle simulation environment. The benchmark evaluates tool use and memory by comparing the post-action environment state with a predefined target state, enabling objective and reproducible evaluation without LLM-based or human scoring. VehicleMemBench includes 23 tool modules, and each sample contains over 80 historical memory events. Experiments show that powerful models perform well on direct instruction tasks but struggle in scenarios involving memory evolution, particularly when user preferences change dynamically. Even advanced memory systems struggle to handle domain-specific memory requirements in this environment. These findings highlight the need for more robust and specialized memory management mechanisms to support long-term adaptive decision-making in real-world in-vehicle systems. To facilitate future research, we release the data and code.