Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment

TL;DR

This paper introduces H$^2$-EMV, a hierarchical episodic memory framework for robots that learns to forget irrelevant details through user interaction, improving scalability and personalization in long-term deployment.

Contribution

It presents a novel hierarchical episodic memory system that uses language models and user feedback to selectively forget, enabling scalable and personalized robot memory.

Findings

Memory size reduced by 45% without loss of QA accuracy.

Query-time compute decreased by 35%.

Question-answering accuracy improved by 70% after adaptation.

Abstract

Robots must verbalize their past experiences when users ask "Where did you put my keys?" or "Why did the task fail?" Yet maintaining life-long episodic memory (EM) from continuous multimodal perception quickly exceeds storage limits and makes real-time query impractical, calling for selective forgetting that adapts to users' notions of relevance. We present H$^2$-EMV, a framework enabling humanoids to learn what to remember through user interaction. Our approach incrementally constructs hierarchical EM, selectively forgets using language-model-based relevance estimation conditioned on learned natural-language rules, and updates these rules given user feedback about forgotten details. Evaluations on simulated household tasks and 20.5-hour-long real-world recordings from ARMAR-7 demonstrate that H$^2$-EMV maintains question-answering accuracy while reducing memory size by 45% and query-time compute by 35%. Critically, performance improves over time - accuracy increases 70% in second-round queries by adapting to user-specific priorities - demonstrating that learned forgetting enables scalable, personalized EM for long-term human-robot collaboration.