Towards a Vision-Language Episodic Memory Framework: Large-scale Pretrained Model-Augmented Hippocampal Attractor Dynamics

TL;DR

This paper introduces VLEM, a novel framework combining large-scale pretrained models with hippocampal attractor dynamics to improve episodic memory modeling in AI and neuroscience, emphasizing interpretability and real-world applicability.

Contribution

It presents a new vision-language episodic memory framework that integrates pretrained models with hippocampal dynamics and introduces EpiGibson for data generation.

Findings

Efficient learning of high-level temporal representations.

Robustness and interpretability demonstrated.

Applicable to real-world scenarios.

Abstract

Modeling episodic memory (EM) remains a significant challenge in both neuroscience and AI, with existing models either lacking interpretability or struggling with practical applications. This paper proposes the Vision-Language Episodic Memory (VLEM) framework to address these challenges by integrating large-scale pretrained models with hippocampal attractor dynamics. VLEM leverages the strong semantic understanding of pretrained models to transform sensory input into semantic embeddings as the neocortex, while the hippocampus supports stable memory storage and retrieval through attractor dynamics. In addition, VLEM incorporates prefrontal working memory and the entorhinal gateway, allowing interaction between the neocortex and the hippocampus. To facilitate real-world applications, we introduce EpiGibson, a 3D simulation platform for generating episodic memory data. Experimental results demonstrate the VLEM framework's ability to efficiently learn high-level temporal representations from sensory input, showcasing its robustness, interpretability, and applicability in real-world scenarios.