Learning to Learn from Multimodal Experience

TL;DR

This paper introduces a novel paradigm where agents learn to adaptively structure and utilize multimodal experience, improving performance and generalization in complex environments.

Contribution

It proposes a learnable memory framework that dynamically organizes multimodal experience based on task needs, moving beyond fixed memory schemas.

Findings

Adaptive memory design improves agent performance across tasks

Learned memory structures enhance generalization in multimodal environments

Dynamic memory organization outperforms static schemas

Abstract

Experience-driven learning has emerged as a promising paradigm for enabling agents to improve from interaction trajectories by accumulating and reusing past experience. However, existing approaches are predominantly developed in textual settings and rely on manually designed memory schemas, limiting their applicability to multimodal environments. In real-world scenarios, experience is inherently multimodal, involving heterogeneous signals across perception, reasoning, and action, which makes effective memory design significantly more challenging. In particular, the optimal way to structure and utilize multimodal experience is highly task-dependent and evolves over time, rendering fixed memory designs insufficient. In this work, we propose a new paradigm, learning to learn from multimodal experience, which shifts memory design from a predefined component to an adaptive and learnable process. Our framework enables agents to dynamically construct, organize, and utilize memory based on task requirements and interaction history, effectively learning how to structure experience for improved performance. Experiments demonstrate that adaptive memory design substantially enhances agent performance and generalization across multimodal tasks, highlighting the critical role of learning memory mechanisms in experience-driven learning.