SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

TL;DR

SMMBench is a new benchmark designed to evaluate multimodal agents' ability to retrieve and reason over evidence scattered across multiple heterogeneous sources, addressing a key gap in current multimodal memory research.

Contribution

The paper introduces SMMBench, a benchmark for source-distributed multimodal memory reasoning, highlighting its importance and providing initial baseline evaluations.

Findings

Current systems struggle with source-distributed multimodal reasoning

SMMBench contains 1877 samples from 264 sources

Baseline models show significant room for improvement in this task

Abstract

Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory, especially when relevant evidence is fragmented across heterogeneous artifacts such as conversations, profiles, screenshots, tables, images, and documents. To address this gap, we introduce Source-distributed Multimodal Memory Benchmark(SMMBench), which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources rather than reason within a single curated context. SMMBench evaluates four core capabilities: (1) cross-source multimodal reasoning; (2) conflict resolution; (3) preference reasoning; (4) memory-grounded action prediction. The benchmark contains 1877 samples grounded in 264 sources. Experiments on representative memory-style and retrieval-based baselines show that current systems still struggle on these capabilities, positioning source-distributed multimodal memory as an important and still under-evaluated challenge for multimodal agents. Our data are available at https://huggingface.co/datasets/HuacanChai/SMMBench.