AR Secretary Agent: Real-time Memory Augmentation via LLM-powered Augmented Reality Glasses

TL;DR

This paper presents an AR Secretary Agent that uses LLMs and computer vision to provide real-time memory augmentation through AR glasses, helping professionals recall personal interactions more effectively.

Contribution

We developed a novel AR system integrating LLMs and computer vision for real-time memory assistance, demonstrating its effectiveness in a user study.

Findings

Memory recall improved by up to 20% in user tests.

System successfully identifies individuals and summarizes past interactions.

Real-time assistance enhances social interaction experiences.

Abstract

Interacting with a significant number of individuals on a daily basis is commonplace for many professionals, which can lead to challenges in recalling specific details: Who is this person? What did we talk about last time? The advant of augmented reality (AR) glasses, equipped with visual and auditory data capture capabilities, presents a solution. In our work, we implemented an AR Secretary Agent with advanced Large Language Models (LLMs) and Computer Vision technologies. This system could discreetly provide real-time information to the wearer, identifying who they are conversing with and summarizing previous discussions. To verify AR Secretary, we conducted a user study with 13 participants and showed that our technique can efficiently help users to memorize events by up to 20\% memory enhancement on our study.