Human-Inspired Context-Selective Multimodal Memory for Social Robots

TL;DR

This paper introduces a human-inspired, context-selective multimodal memory system for social robots that enhances personalized, natural interactions by prioritizing emotionally salient and novel experiences.

Contribution

It presents a novel memory architecture that combines multimodal episodic recall with context-based prioritization, inspired by cognitive neuroscience, for improved social robot interactions.

Findings

Achieved a Spearman correlation of 0.506 in memory selectivity, surpassing human consistency.

Improved Recall@1 by up to 13% in multimodal retrieval tasks.

Maintains real-time performance in social scenarios.

Abstract

Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the context. However, most current social robots and embodied agents rely on non-selective, text-based memory, limiting their ability to support personalized, context-aware interactions. Drawing inspiration from cognitive neuroscience, we propose a context-selective, multimodal memory architecture for social robots that captures and retrieves both textual and visual episodic traces, prioritizing moments characterized by high emotional salience or scene novelty. By associating these memories with individual users, our system enables socially personalized recall and more natural, grounded dialogue. We evaluate the selective storage mechanism using a curated dataset of social scenarios, achieving a Spearman correlation of 0.506, surpassing human consistency ($\rho=0.415$) and outperforming existing image memorability models. In multimodal retrieval experiments, our fusion approach improves Recall@1 by up to 13\% over unimodal text or image retrieval. Runtime evaluations confirm that the system maintains real-time performance. Qualitative analyses further demonstrate that the proposed framework produces richer and more socially relevant responses than baseline models. This work advances memory design for social robots by bridging human-inspired selectivity and multimodal retrieval to enhance long-term, personalized human-robot interaction.