CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only

TL;DR

This paper introduces CAAP, a novel prompt strategy for LLM-based agents that solve computer tasks using only GUI screenshots and keyboard/mouse actions, achieving high success rates without environment-specific APIs.

Contribution

The paper presents CAAP, a context-aware prompting method enabling LLM agents to perform GUI-based tasks solely from screenshots, eliminating the need for environment-specific code or large demonstration datasets.

Findings

Achieved 94.5% success rate on MiniWoB++

Outperformed previous image-based agents on WebShop

Demonstrated potential for cross-application task automation

Abstract

Software robots have long been used in Robotic Process Automation (RPA) to automate mundane and repetitive computer tasks. With the advent of Large Language Models (LLMs) and their advanced reasoning capabilities, these agents are now able to handle more complex or previously unseen tasks. However, LLM-based automation techniques in recent literature frequently rely on HTML source code for input or application-specific API calls for actions, limiting their applicability to specific environments. We propose an LLM-based agent that mimics human behavior in solving computer tasks. It perceives its environment solely through screenshot images, which are then converted into text for an LLM to process. By leveraging the reasoning capability of the LLM, we eliminate the need for large-scale human demonstration data typically required for model training. The agent only executes keyboard and mouse operations on Graphical User Interface (GUI), removing the need for pre-provided APIs to function. To further enhance the agent's performance in this setting, we propose a novel prompting strategy called Context-Aware Action Planning (CAAP) prompting, which enables the agent to thoroughly examine the task context from multiple perspectives. Our agent achieves an average success rate of 94.5% on MiniWoB++ and an average task score of 62.3 on WebShop, outperforming all previous studies of agents that rely solely on screen images. This method demonstrates potential for broader applications, particularly for tasks requiring coordination across multiple applications on desktops or smartphones, marking a significant advancement in the field of automation agents. Codes and models are accessible at https://github.com/caap-agent/caap-agent.