TL;DR
LlamaPIE is a real-time, proactive in-ear conversation assistant that discreetly enhances human dialogue by predicting needs and providing concise guidance without interrupting, using on-device processing and a two-model pipeline.
Contribution
This work introduces LlamaPIE, the first proactive hearable assistant operating in real-time with on-device processing and a novel two-model approach for context-aware, unobtrusive assistance.
Findings
Effective in real-world datasets
User preference for proactive assistance
On-device processing enables real-time operation
Abstract
We introduce LlamaPIE, the first real-time proactive assistant designed to enhance human conversations through discreet, concise guidance delivered via hearable devices. Unlike traditional language models that require explicit user invocation, this assistant operates in the background, anticipating user needs without interrupting conversations. We address several challenges, including determining when to respond, crafting concise responses that enhance conversations, leveraging knowledge of the user for context-aware assistance, and real-time, on-device processing. To achieve this, we construct a semi-synthetic dialogue dataset and propose a two-model pipeline: a small model that decides when to respond and a larger model that generates the response. We evaluate our approach on real-world datasets, demonstrating its effectiveness in providing helpful, unobtrusive assistance. User…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
