HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering

TL;DR

This paper introduces a novel non-parametric memory framework for embodied agents that explicitly separates episodic and semantic memories, improving long-term exploration and question answering in complex environments.

Contribution

It presents a retrieval-first, reasoning-assisted memory system and a program-style rule extraction method, enabling better reuse and generalization of experiences in embodied AI.

Findings

Achieved state-of-the-art results on A-EQA and GOAT-Bench benchmarks.

Episodic memory enhances exploration efficiency.

Semantic memory improves complex reasoning capabilities.

Abstract

Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.