Intelligent Virtual Assistants with LLM-based Process Automation
Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua, Song, Longfei Li, Jinjie Gu, Chenyi Zhuang

TL;DR
This paper introduces a novel LLM-based virtual assistant system capable of executing multi-step mobile app tasks from natural language commands, demonstrating real-world deployment and improved automation capabilities.
Contribution
It presents the first real-world deployment of an LLM-based virtual assistant for mobile app automation with a comprehensive architecture and methodology.
Findings
Successfully completed complex tasks in Alipay using natural language instructions.
Demonstrated robustness and effectiveness of LLMs in real-world mobile app automation.
Achieved end-to-end automation with high accuracy in multi-step process execution.
Abstract
While intelligent virtual assistants like Siri, Alexa, and Google Assistant have become ubiquitous in modern life, they still face limitations in their ability to follow multi-step instructions and accomplish complex goals articulated in natural language. However, recent breakthroughs in large language models (LLMs) show promise for overcoming existing barriers by enhancing natural language processing and reasoning capabilities. Though promising, applying LLMs to create more advanced virtual assistants still faces challenges like ensuring robust performance and handling variability in real-world user commands. This paper proposes a novel LLM-based virtual assistant that can automatically perform multi-step operations within mobile apps based on high-level user requests. The system represents an advance in assistants by providing an end-to-end solution for parsing instructions, reasoning…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Business Process Modeling and Analysis · Robotic Process Automation Applications
MethodsBalanced Selection
