Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation

TL;DR

This paper presents MobileGPT, a human-like memory-enhanced LLM system for mobile task automation that improves accuracy, adaptability, and reduces latency and costs.

Contribution

Introducing MobileGPT, a novel LLM-based mobile task automator with human-like app memory for improved learning and adaptation.

Findings

Achieves 82.7% task automation accuracy

Adapts tasks with 98.75% accuracy in different contexts

Reduces latency and cost by over 60%

Abstract

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app -- explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.