Battle Ground: Data Collection and Labeling of CTF Games to Understand Human Cyber Operators

TL;DR

This paper introduces a methodology for collecting and labeling data from realistic cyber CTF games to analyze human attacker and defender behavior, using the MITRE ATT&CK framework and a new tool called Pathfinder.

Contribution

It presents a systematic approach for analyzing human cyber operations during CTFs, including data collection, labeling, and preliminary analysis of keystroke accuracy and action frequency.

Findings

Keystroke accuracy correlates with game score outcomes.

Action classification within MITRE ATT&CK reveals behavioral patterns.

Mathematical trends suggest potential for behavioral modeling.

Abstract

Industry standard frameworks are now widespread for labeling the high-level stages and granular actions of attacker and defender behavior in cyberspace. While these labels are used for atomic actions, and to some extent for sequences of actions, there remains a need for labeled data from realistic full-scale attacks. This data is valuable for better understanding human actors' decisions, behaviors, and individual attributes. The analysis could lead to more effective attribution and disruption of attackers. We present a methodological approach and exploratory case study for systematically analyzing human behavior during a cyber offense/defense capture-the-flag (CTF) game. We describe the data collection and analysis to derive a metric called keystroke accuracy. After collecting players' commands, we label them using the MITRE ATT&CK framework using a new tool called Pathfinder. We present results from preliminary analysis of participants' keystroke accuracy and its relation to score outcome in CTF games. We describe frequency of action classification within the MITRE ATT&CK framework and discuss some of the mathematical trends suggested by our observations. We conclude with a discussion of extensions for the methodology, including performance evaluation during games and the potential use of this methodology for training artificial intelligence.