Can Language Models Pass Software Testing Certification Exams? a case study

TL;DR

This study evaluates whether large language models can pass software testing certification exams, analyzing their understanding, reasoning, and performance across different question types and transformations.

Contribution

It provides a comprehensive assessment of 60 LLMs on ISTQB exams, revealing their capabilities and limitations in software testing knowledge and reasoning.

Findings

Two models passed all certification exams with at least 65% score.

Commercial models generally outperform open-source models.

Transformations affect models' ability to answer correctly.

Abstract

Large Language Models (LLMs) play a pivotal role in both academic research and broader societal applications. LLMs are increasingly used in software testing activities such as test case generation, selection, and repair. However, several important questions remain: (1) do LLMs possess enough information about software testing principles to perform software testing tasks effectively? (2) do LLMs possess sufficient conceptual understanding of software testing to answer software testing questions under metamorphic transformations? and (3) do certain properties of software testing questions influence the performance of LLMs? To answer these questions, this study evaluates 60 multimodal language models from both commercial vendors and the open-source community. The evaluation is performed using 30 sample exams of different types (core foundation, core advanced, specialist, and expert) from the International Software Testing Qualifications Board (ISTQB), which are used to assess the competence of human testers. In total, each model is evaluated on 1,171 questions. Furthermore, to ensure sufficient conceptual understanding, the models are also tested on exam questions transformed using context-preserving metamorphic techniques. Two models passed all the certifications by scoring at least 65% in all of the 30 certification exams, with commercial models generally outperforming open-source ones. We analyze the reasons behind incorrect answers and provide recommendations for improving the design of software testing certification exams.