Challenges Arising from Prerequisite Testing in Cybersecurity Games

TL;DR

This study investigates how prerequisite testing, including quizzes and self-assessment, can predict learner performance in cybersecurity games, highlighting the importance of well-designed pretests for effective learning and engagement.

Contribution

The paper presents an experiment demonstrating that a short, well-designed quiz can accurately predict cybersecurity game performance, addressing challenges in pretest design.

Findings

Quiz predicts performance accurately

Self-assessment does not predict performance

Designing effective pretests involves calibration and integration

Abstract

Cybersecurity games are an attractive and popular method of active learning. However, the majority of current games are created for advanced players, which often leads to frustration in less experienced learners. Therefore, we decided to focus on a diagnostic assessment of participants entering the games. We assume that information about the players' knowledge, skills, and experience enables tutors or learning environments to suitably assist participants with game challenges and maximize learning in their virtual adventure. In this paper, we present a pioneering experiment examining the predictive value of a short quiz and self-assessment for identifying learners' readiness before playing a cybersecurity game. We hypothesized that these predictors would model players' performance. A linear regression analysis showed that the game performance can be accurately predicted by well-designed prerequisite testing, but not by self-assessment. At the same time, we identified major challenges related to the design of pretests for cybersecurity games: calibrating test questions with respect to the skills relevant for the game, minimizing the quiz's length while maximizing its informative value, and embedding the pretest in the game. Our results are relevant for educational researchers and cybersecurity instructors of students at all learning levels.