GUIDE: Graphical User Interface Data for Execution

TL;DR

GUIDE is a comprehensive dataset designed to enhance multimodal large language models for robotic process automation across diverse websites and platforms, enabling more effective automation and understanding of graphical user interfaces.

Contribution

We introduce GUIDE, a new multi-platform GUI dataset with detailed annotations, and develop V-Zen, the first RPA model leveraging this dataset for multi-website automation.

Findings

GUIDE covers diverse websites and platforms.

V-Zen demonstrates effective multi-website automation.

Dataset facilitates research in GUI-based LLM applications.

Abstract

In this paper, we introduce GUIDE, a novel dataset tailored for the advancement of Multimodal Large Language Model (MLLM) applications, particularly focusing on Robotic Process Automation (RPA) use cases. Our dataset encompasses diverse data from various websites including Apollo(62.67\%), Gmail(3.43\%), Calendar(10.98\%) and Canva(22.92\%). Each data entry includes an image, a task description, the last action taken, CoT and the next action to be performed along with grounding information of where the action needs to be executed. The data is collected using our in-house advanced annotation tool NEXTAG (Next Action Grounding and Annotation Tool). The data is adapted for multiple OS, browsers and display types. It is collected by multiple annotators to capture the variation of design and the way person uses a website. Through this dataset, we aim to facilitate research and development in the realm of LLMs for graphical user interfaces, particularly in tasks related to RPA. The dataset's multi-platform nature and coverage of diverse websites enable the exploration of cross-interface capabilities in automation tasks. We believe that our dataset will serve as a valuable resource for advancing the capabilities of multi-platform LLMs in practical applications, fostering innovation in the field of automation and natural language understanding. Using GUIDE, we build V-Zen, the first RPA model to automate multiple websites using our in-House Automation tool AUTONODE