CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only
Junhee Cho, Jihoon Kim, Daseul Bae, Jinho Choo, Youngjune Gwon,, Yeong-Dae Kwon

TL;DR
This paper introduces CAAP, a novel prompt strategy for LLM-based agents that solve computer tasks using only GUI screenshots and keyboard/mouse actions, achieving high success rates without environment-specific APIs.
Contribution
The paper presents CAAP, a context-aware prompting method enabling LLM agents to perform GUI-based tasks solely from screenshots, eliminating the need for environment-specific code or large demonstration datasets.
Findings
Achieved 94.5% success rate on MiniWoB++
Outperformed previous image-based agents on WebShop
Demonstrated potential for cross-application task automation
Abstract
Software robots have long been used in Robotic Process Automation (RPA) to automate mundane and repetitive computer tasks. With the advent of Large Language Models (LLMs) and their advanced reasoning capabilities, these agents are now able to handle more complex or previously unseen tasks. However, LLM-based automation techniques in recent literature frequently rely on HTML source code for input or application-specific API calls for actions, limiting their applicability to specific environments. We propose an LLM-based agent that mimics human behavior in solving computer tasks. It perceives its environment solely through screenshot images, which are then converted into text for an LLM to process. By leveraging the reasoning capability of the LLM, we eliminate the need for large-scale human demonstration data typically required for model training. The agent only executes keyboard and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsContext-Aware Activity Recognition Systems · Real-Time Systems Scheduling · Robotics and Automated Systems
MethodsSparse Evolutionary Training
