Visual Feedback for Players of Multi-Level Capture the Flag Games: Field Usability Study

TL;DR

This study develops and evaluates interactive visualizations to provide post-game feedback in cybersecurity Capture the Flag games, enhancing learning and motivation through user-centered design and field usability testing.

Contribution

It introduces two novel interactive visualizations tailored for cybersecurity game feedback, validated through a real-world usability study with game participants.

Findings

Visualizations are effective for instant post-game feedback.

Participants found visualizations adequate but showed no strong preference.

Field study confirms usability in a real training context.

Abstract

Capture the Flag games represent a popular method of cybersecurity training. Providing meaningful insight into the training progress is essential for increasing learning impact and supporting participants' motivation, especially in advanced hands-on courses. In this paper, we investigate how to provide valuable post-game feedback to players of serious cybersecurity games through interactive visualizations. In collaboration with domain experts, we formulated user requirements that cover three cognitive perspectives: gameplay overview, person-centric view, and comparative feedback. Based on these requirements, we designed two interactive visualizations that provide complementary views on game results. They combine a known clustering and time-based visual approaches to show game results in a way that is easy to decode for players. The purposefulness of our visual feedback was evaluated in a usability field study with attendees of the Summer School in Cyber Security. The evaluation confirmed the adequacy of the two visualizations for instant post-game feedback. Despite our initial expectations, there was no strong preference for neither of the visualizations in solving different tasks.