Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU

TL;DR

This paper introduces a privacy-preserving, real-time conversational assistant for procedural tasks using audio and IMU data, featuring a novel finetuning method to improve dialogue relevance and efficiency.

Contribution

It presents a lightweight, edge-deployable assistant for manual tasks, with a new finetuning approach to enhance dialogue quality and reduce computational load.

Findings

>30% improvement in F-score for dialogue relevance

16x faster inference after finetuning

Effective on edge devices without cloud dependence

Abstract

Real-time conversational assistants for procedural tasks often depend on video input, which can be computationally expensive and compromise user privacy. For the first time, we propose a real-time conversational assistant that provides comprehensive guidance for a procedural task using only lightweight privacy-preserving modalities such as audio and IMU inputs from a user's wearable device to understand the context. This assistant proactively communicates step-by-step instructions to a user performing a furniture assembly task, and answers user questions. We construct a dataset containing conversations where the assistant guides the user in performing the task. On observing that an off-the-shelf language model is a very talkative assistant, we design a novel User Whim Agnostic (UWA) LoRA finetuning method which improves the model's ability to suppress less informative dialogues, while maintaining its tendency to communicate important instructions. This leads to >30% improvement in the F-score. Finetuning the model also results in a 16x speedup by eliminating the need to provide in-context examples in the prompt. We further describe how such an assistant is implemented on edge devices with no dependence on the cloud.