Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark

TL;DR

This paper explores how to optimize LLM-based offensive security agents for CTF challenges by tuning hyperparameters, using LLM as a judge, and introducing a lightweight benchmark dataset.

Contribution

It introduces CTFJudge, a framework for evaluating agents, a new metric CTF Competency Index, and a curated benchmark dataset CTFTiny, advancing research in cybersecurity LLM agents.

Findings

Optimal hyperparameter settings improve agent performance.

LLM as a judge provides detailed analysis of agent trajectories.

CTFTiny enables rapid evaluation across diverse CTF challenges.

Abstract

Recent advances in LLM agentic systems have improved the automation of offensive security tasks, particularly for Capture the Flag (CTF) challenges. We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents. First, we present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps. Second, we propose a novel metric, CTF Competency Index (CCI) for partial correctness, revealing how closely agent solutions align with human-crafted gold standards. Third, we examine how LLM hyperparameters, namely temperature, top-p, and maximum token length, influence agent performance and automated cybersecurity task planning. For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography. Our findings identify optimal multi-agent coordination settings and lay the groundwork for future LLM agent research in cybersecurity. We make CTFTiny open source to public https://github.com/NYU-LLM-CTF/CTFTiny along with CTFJudge on https://github.com/NYU-LLM-CTF/CTFJudge.