Intelligent Virtual Assistants with LLM-based Process Automation

TL;DR

This paper introduces a novel LLM-based virtual assistant system capable of executing multi-step mobile app tasks from natural language commands, demonstrating real-world deployment and improved automation capabilities.

Contribution

It presents the first real-world deployment of an LLM-based virtual assistant for mobile app automation with a comprehensive architecture and methodology.

Findings

Successfully completed complex tasks in Alipay using natural language instructions.

Demonstrated robustness and effectiveness of LLMs in real-world mobile app automation.

Achieved end-to-end automation with high accuracy in multi-step process execution.

Abstract

While intelligent virtual assistants like Siri, Alexa, and Google Assistant have become ubiquitous in modern life, they still face limitations in their ability to follow multi-step instructions and accomplish complex goals articulated in natural language. However, recent breakthroughs in large language models (LLMs) show promise for overcoming existing barriers by enhancing natural language processing and reasoning capabilities. Though promising, applying LLMs to create more advanced virtual assistants still faces challenges like ensuring robust performance and handling variability in real-world user commands. This paper proposes a novel LLM-based virtual assistant that can automatically perform multi-step operations within mobile apps based on high-level user requests. The system represents an advance in assistants by providing an end-to-end solution for parsing instructions, reasoning about goals, and executing actions. LLM-based Process Automation (LLMPA) has modules for decomposing instructions, generating descriptions, detecting interface elements, predicting next actions, and error checking. Experiments demonstrate the system completing complex mobile operation tasks in Alipay based on natural language instructions. This showcases how large language models can enable automated assistants to accomplish real-world tasks. The main contributions are the novel LLMPA architecture optimized for app process automation, the methodology for applying LLMs to mobile apps, and demonstrations of multi-step task completion in a real-world environment. Notably, this work represents the first real-world deployment and extensive evaluation of a large language model-based virtual assistant in a widely used mobile application with an enormous user base numbering in the hundreds of millions.