The Power of Words: Generating PowerShell Attacks from Natural Language

TL;DR

This paper demonstrates how neural machine translation models can be fine-tuned to automatically generate offensive PowerShell scripts from natural language descriptions, revealing new AI capabilities in cybersecurity contexts.

Contribution

It introduces two novel datasets for training and evaluating NMT models on PowerShell code and natural language descriptions, and provides an extensive analysis of model performance.

Findings

Fine-tuned NMT models effectively generate offensive PowerShell code.

Our datasets improve the training and evaluation of code generation models.

Specialized models outperform general LLMs like ChatGPT in this domain.

Abstract

As the Windows OS stands out as one of the most targeted systems, the PowerShell language has become a key tool for malicious actors and cybersecurity professionals (e.g., for penetration testing). This work explores an uncharted domain in AI code generation by automatically generating offensive PowerShell code from natural language descriptions using Neural Machine Translation (NMT). For training and evaluation purposes, we propose two novel datasets with PowerShell code samples, one with manually curated descriptions in natural language and another code-only dataset for reinforcing the training. We present an extensive evaluation of state-of-the-art NMT models and analyze the generated code both statically and dynamically. Results indicate that tuning NMT using our dataset is effective at generating offensive PowerShell code. Comparative analysis against the most widely used LLM service ChatGPT reveals the specialized strengths of our fine-tuned models.