RUM: Rule+LLM-Based Comprehensive Assessment on Testing Skills

TL;DR

RUM is a novel assessment system combining rules and large language models to evaluate both objective and subjective testing skills, significantly improving efficiency and accuracy over traditional methods.

Contribution

It introduces a comprehensive testing skill assessment approach that integrates rule-based and LLM-based analysis, addressing limitations of prior objective-only systems.

Findings

Assessment efficiency increased by 80.77%

Assessment costs reduced by 97.38%

Maintains high accuracy and consistency

Abstract

Over the past eight years, the META method has served as a multidimensional testing skill assessment system in the National College Student Contest on Software Testing, successfully assessing over 100,000 students' testing skills. However, META is primarily limited to the objective assessment of test scripts, lacking the ability to automatically assess subjective aspects such as test case and test report. To address this limitation, this paper proposes RUM, a comprehensive assessment approach that combines rules and large language models (LLMs). RUM achieves a comprehensive assessment by rapidly processing objective indicators through rules while utilizing LLMs for in-depth subjective analysis of test case documents, test scripts, and test reports. The experimental results show that compared to traditional manual testing skill assessment, RUM improves assessment efficiency by 80.77\% and reduces costs by 97.38\%, while maintaining high accuracy and consistency of assessment. By applying RUM on the contest on software testing, we find that it not only enhances the efficiency and scalability of skill assessment in software testing education, but also provides teachers with more comprehensive and objective evidence for student ability assessment, facilitating personalized teaching and learning. This study offers new insights into the assessment of testing skills, which are expected to promote further development in test process optimization and software quality assurance.