Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions
Thanapong Intharah, Daniyar Turmukhambetov, Gabriel J. Brostow

TL;DR
This paper introduces HILC, a user-in-the-loop system that enables non-programmers to create automation scripts through demonstration and follow-up questions, effectively handling complex GUI tasks.
Contribution
The paper presents a novel demonstration-based framework with follow-up queries that improves script generation for GUI automation by non-programmers.
Findings
HILC outperforms Sikuli Slides in user study tasks.
Users can efficiently teach HILC to perform complex GUI tasks.
HILC requires less training time compared to baseline systems.
Abstract
Non-programming users should be able to create their own customized scripts to perform computer-based tasks for them, just by demonstrating to the machine how it's done. To that end, we develop a system prototype which learns-by-demonstration called HILC (Help, It Looks Confusing). Users train HILC to synthesize a task script by demonstrating the task, which produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. While pure programming-by-demonstration is still unrealistic, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system. Our models of events and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Visualization and Analytics · Video Analysis and Summarization · Human Pose and Action Recognition
