OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
Lingkai Meng, Yu Shao, Long Yuan, Longbin Lai, Peng Cheng, Wenyuan Yu, Wenjie Zhang, Xuemin Lin, and Jingren Zhou

TL;DR
OSS-UAgent is an automated framework using large language models to evaluate open source software usability by simulating developer interactions and assessing code quality across multiple dimensions.
Contribution
It introduces a novel agent-based, LLM-powered framework for scalable, automated usability evaluation of OSS, addressing limitations of traditional human-based methods.
Findings
Effective in evaluating graph analytics platforms
Automates multiple usability dimensions
Simulates developers of various experience levels
Abstract
Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsUsability and User Interface Design · Software Engineering Techniques and Practices · Software Engineering Research
