OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software

TL;DR

OSS-UAgent is an automated framework using large language models to evaluate open source software usability by simulating developer interactions and assessing code quality across multiple dimensions.

Contribution

It introduces a novel agent-based, LLM-powered framework for scalable, automated usability evaluation of OSS, addressing limitations of traditional human-based methods.

Findings

Effective in evaluating graph analytics platforms

Automates multiple usability dimensions

Simulates developers of various experience levels

Abstract

Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation.