Dynamic Planning for LLM-based Graphical User Interface Automation

TL;DR

This paper introduces D-PoT, a dynamic planning approach for LLM-based GUI agents that adapts plans based on environmental feedback, significantly improving task accuracy over existing methods.

Contribution

The paper proposes D-PoT, a novel dynamic planning method that enhances LLM-based GUI automation by adapting to feedback and reducing hallucinations, outperforming previous approaches.

Findings

D-PoT achieves +12.7% accuracy over GPT-4V baseline.

Dynamic planning improves adaptability to unseen tasks.

D-PoT reduces hallucinations in LLM-based GUI agents.

Abstract

The advent of large language models (LLMs) has spurred considerable interest in advancing autonomous LLMs-based agents, particularly in intriguing applications within smartphone graphical user interfaces (GUIs). When presented with a task goal, these agents typically emulate human actions within a GUI environment until the task is completed. However, a key challenge lies in devising effective plans to guide action prediction in GUI tasks, though planning have been widely recognized as effective for decomposing complex tasks into a series of steps. Specifically, given the dynamic nature of environmental GUIs following action execution, it is crucial to dynamically adapt plans based on environmental feedback and action history.We show that the widely-used ReAct approach fails due to the excessively long historical dialogues. To address this challenge, we propose a novel approach called Dynamic Planning of Thoughts (D-PoT) for LLM-based GUI agents.D-PoT involves the dynamic adjustment of planning based on the environmental feedback and execution history. Experimental results reveal that the proposed D-PoT significantly surpassed the strong GPT-4V baseline by +12.7% (34.66% $\rightarrow$ 47.36%) in accuracy. The analysis highlights the generality of dynamic planning in different backbone LLMs, as well as the benefits in mitigating hallucinations and adapting to unseen tasks. Code is available at https://github.com/sqzhang-lazy/D-PoT.