GazeNoter: Co-Piloted AR Note-Taking via Gaze Selection of LLM Suggestions to Match Users' Intentions

TL;DR

GazeNoter is an AR-based note-taking system that uses gaze to select LLM-generated suggestions, enabling real-time, user-aligned notes during speeches and discussions, reducing cognitive load and increasing interaction efficiency.

Contribution

The paper introduces GazeNoter, a novel AR system that integrates gaze-based selection with LLM suggestions for real-time, user-in-the-loop note-taking in speech and discussion settings.

Findings

GazeNoter is effective in static and mobile scenarios.

Users can quickly select and customize notes via gaze.

The system improves note relevance and user engagement.

Abstract

Note-taking is critical during speeches and discussions, serving not only for later summarization and organization but also for real-time question and opinion reminding in question-and-answer sessions or timely contributions in discussions. Manually typing on smartphones for note-taking could be distracting and increase cognitive load for users. While large language models (LLMs) are used to automatically generate summaries and highlights, the content generated by artificial intelligence (AI) may not match users' intentions without user input or interaction. Therefore, we propose an AI-copiloted augmented reality (AR) system, GazeNoter, to allow users to swiftly select diverse LLM-generated suggestions via gaze on an AR headset for real-time note-taking. GazeNoter leverages an AR headset as a medium for users to swiftly adjust the LLM output to match their intentions, forming a user-in-the-loop AI system for both within-context and beyond-context notes. We conducted two user studies to verify the usability of GazeNoter in attending speeches in a static sitting condition and walking meetings and discussions in a mobile walking condition, respectively.