Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
Jiateng Liu, Rushi Wang, Bingxuan Li, Kunlun Zhu, Yifan Shen, Qingyun Wang, Ahmed Abbasi, Denghui Zhang, Heng Ji

TL;DR
This paper explores how augmenting usability heuristics can enhance the robustness and reliability of computer-use agents across evolving interfaces, combining interface design improvements with agent capabilities.
Contribution
It introduces augmented heuristics based on Nielsen's principles, evaluated through controlled environments and human studies, to improve agent task success and efficiency.
Findings
Augmented heuristics improve task completion rates.
Design augmentations modestly increase efficiency.
Human studies show no usability regressions.
Abstract
Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent compatible interface design offers a complementary path by aligning interaction semantics with agent prior knowledge. In this paper, we revisit Nielsen 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
