Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experience

TL;DR

This paper introduces a hierarchical memory structure and leverages large language models to verbalize and query a robot's lifelong experiences, enhancing human-robot interaction with scalable and flexible summarization.

Contribution

It presents a novel hierarchical episodic memory representation combined with large language models for verbalization and querying of lifelong robot experiences.

Findings

Effective verbalization of long-term robot data

Scalable approach with low computational costs

Flexible application across different data sources

Abstract

Verbalization of robot experience, i.e., summarization of and question answering about a robot's past, is a crucial ability for improving human-robot interaction. Previous works applied rule-based systems or fine-tuned deep models to verbalize short (several-minute-long) streams of episodic data, limiting generalization and transferability. In our work, we apply large pretrained models to tackle this task with zero or few examples, and specifically focus on verbalizing life-long experiences. For this, we derive a tree-like data structure from episodic memory (EM), with lower levels representing raw perception and proprioception data, and higher levels abstracting events to natural language concepts. Given such a hierarchical representation built from the experience stream, we apply a large language model as an agent to interactively search the EM given a user's query, dynamically expanding (initially collapsed) tree nodes to find the relevant information. The approach keeps computational costs low even when scaling to months of robot experience data. We evaluate our method on simulated household robot data, human egocentric videos, and real-world robot recordings, demonstrating its flexibility and scalability.