GPA: Learning GUI Process Automation from Demonstrations
Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan Li

TL;DR
GPA is a vision-based GUI automation method that offers robustness, reliability, and privacy, enabling fast, stable process replay from a single demonstration, outperforming existing solutions in success rate and speed.
Contribution
GPA introduces a novel approach combining Monte Carlo localization and readiness calibration for robust, deterministic, and private GUI process automation from demonstrations.
Findings
GPA achieves higher success rate than Gemini 3 Pro in long-horizon tasks.
GPA is 10 times faster in executing GUI tasks.
GPA enables fast, reliable, and privacy-preserving GUI automation.
Abstract
GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
