GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning

TL;DR

GptVoiceTasker leverages large language models to improve mobile task efficiency by understanding user commands, automating tasks, and learning from user history, resulting in a 34.85% efficiency boost in real-world tests.

Contribution

This paper introduces GptVoiceTasker, a novel virtual assistant that combines LLMs with dynamic learning to enhance mobile task execution and user experience.

Findings

Boosted task efficiency by 34.85% in real-world scenarios

Achieved high accuracy in command interpretation

Open-sourced for further research and development

Abstract

Virtual assistants have the potential to play an important role in helping users achieves different tasks. However, these systems face challenges in their real-world usability, characterized by inefficiency and struggles in grasping user intentions. Leveraging recent advances in Large Language Models (LLMs), we introduce GptVoiceTasker, a virtual assistant poised to enhance user experiences and task efficiency on mobile devices. GptVoiceTasker excels at intelligently deciphering user commands and executing relevant device interactions to streamline task completion. The system continually learns from historical user commands to automate subsequent usages, further enhancing execution efficiency. Our experiments affirm GptVoiceTasker's exceptional command interpretation abilities and the precision of its task automation module. In our user study, GptVoiceTasker boosted task efficiency in real-world scenarios by 34.85%, accompanied by positive participant feedback. We made GptVoiceTasker open-source, inviting further research into LLMs utilization for diverse tasks through prompt engineering and leveraging user usage data to improve efficiency.