Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception

TL;DR

This paper introduces a memory augmentation system using language-encoded egocentric videos stored in a vector database, leveraging large vision-language models for efficient retrieval and significantly improving recall performance.

Contribution

It proposes a novel approach combining natural language encoding and large language models for efficient storage and retrieval of egocentric video data for memory augmentation.

Findings

Achieved state-of-the-art BLEU score of 8.3 on QA-Ego4D dataset.

Participants showed improved recall with the agent in user studies.

Demonstrated practical applicability and user acceptance of the system.

Abstract

We depend on our own memory to encode, store, and retrieve our experiences. However, memory lapses can occur. One promising avenue for achieving memory augmentation is through the use of augmented reality head-mounted displays to capture and preserve egocentric videos, a practice commonly referred to as lifelogging. However, a significant challenge arises from the sheer volume of video data generated through lifelogging, as the current technology lacks the capability to encode and store such large amounts of data efficiently. Further, retrieving specific information from extensive video archives requires substantial computational power, further complicating the task of quickly accessing desired content. To address these challenges, we propose a memory augmentation agent that involves leveraging natural language encoding for video data and storing them in a vector database. This approach harnesses the power of large vision language models to perform the language encoding process. Additionally, we propose using large language models to facilitate natural language querying. Our agent underwent extensive evaluation using the QA-Ego4D dataset and achieved state-of-the-art results with a BLEU score of 8.3, outperforming conventional machine learning models that scored between 3.4 and 5.8. Additionally, we conducted a user study in which participants interacted with the human memory augmentation agent through episodic memory and open-ended questions. The results of this study show that the agent results in significantly better recall performance on episodic memory tasks compared to human participants. The results also highlight the agent's practical applicability and user acceptance.