Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

TL;DR

Prompt2Task is a system that automates smartphone UI tasks from textual prompts by mimicking human cognition, significantly improving success rates and reducing user interventions, with potential applications in tutorials, assistance, and customer service.

Contribution

The paper introduces Prompt2Task, a novel system that interprets textual prompts to generate and execute smartphone UI automation, reducing the need for scripting expertise.

Findings

Success rate increased from 22.28% to 95.24%.

Average user interventions per task reduced to 0.69.

Effective in applications like tutorials and smart assistance.

Abstract

UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28\% success rate in the baseline to 95.24\% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service.