Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus

TL;DR

This paper investigates how large language models can enhance ISTQB-based software testing education by creating a specialized dataset, optimizing prompts, evaluating LLM performance, and providing integration recommendations.

Contribution

It introduces a comprehensive ISTQB-aligned dataset, develops optimized prompts, systematically evaluates LLMs, and offers practical guidance for incorporating LLMs into software testing education.

Findings

Created a dataset with 28 exams and 1,145 questions

Developed prompts that improve LLM accuracy and explanations

Provided actionable recommendations for LLM integration

Abstract

Software testing is a critical component in the software engineering field and is important for software engineering education. Thus, it is vital for academia to continuously improve and update educational methods to reflect the current state of the field. The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia. However, ISTQB-based learning has been rarely applied with recent generative artificial intelligence advances. Despite the growing capabilities of large language models (LLMs), ISTQB-based learning and instruction with LLMs have not been thoroughly explored. This paper explores and evaluates how LLMs can complement the ISTQB framework for higher education. The findings present four key contributions: (i) the creation of a comprehensive ISTQB-aligned dataset spanning over a decade, consisting of 28 sample exams and 1,145 questions; (ii) the development of a domain-optimized prompt that enhances LLM precision and explanation quality on ISTQB tasks; (iii) a systematic evaluation of state-of-the-art LLMs on this dataset; and (iv) actionable insights and recommendations for integrating LLMs into software testing education. These findings highlight the promise of LLMs in supporting ISTQB certification preparation and offer a foundation for their broader use in software engineering at higher education.