Evaluating LLM-Generated Obfuscated XSS Payloads for Machine Learning-Based Detection

TL;DR

This paper develops a pipeline using large language models to generate and evaluate obfuscated XSS payloads based on runtime behavior, revealing current limitations in behavior preservation and detection improvement.

Contribution

It introduces a structured approach combining deterministic transformations and LLMs with runtime evaluation to generate behavior-preserving obfuscated XSS payloads.

Findings

Baseline LLMs achieve 0.15 behavior match rate

Fine-tuning improves match rate to 0.22

Adding generated payloads does not enhance detection performance

Abstract

Cross-site scripting (XSS) remains a persistent web security vulnerability, especially because obfuscation can change the surface form of a malicious payload while preserving its behavior. These transformations make it difficult for traditional and machine learning-based detection systems to reliably identify attacks. Existing approaches for generating obfuscated payloads often emphasize syntactic diversity, but they do not always ensure that the generated samples remain behaviorally valid. This paper presents a structured pipeline for generating and evaluating obfuscated XSS payloads using large language models (LLMs). The pipeline combines deterministic transformation techniques with LLM-based generation and uses a browser- based runtime evaluation procedure to compare payload behavior in a controlled execution environment. This allows generated samples to be assessed through observable runtime behavior rather than syntactic similarity alone. In the evaluation, an untuned baseline language model achieves a runtime behavior match rate of 0.15, while fine-tuning on behavior-preserving source-target obfuscation pairs improves the match rate to 0.22. Although this represents a measurable improvement, the results show that current LLMs still struggle to generate obfuscations that preserve observed runtime behavior. A downstream classifier evaluation further shows that adding generated payloads does not improve detection performance in this setting, although behavior- filtered generated samples can be incorporated without materially degrading performance. Overall, the study demonstrates both the promise and the limits of applying generative models to adversarial security data generation and emphasizes the importance of runtime behavior checks in improving the quality of generated data for downstream detection systems.