Trust, But Verify: An Empirical Evaluation of AI-Generated Code for SDN Controllers

TL;DR

This paper empirically evaluates AI-generated code for SDN controllers, analyzing correctness and functionality across different AI tools and prompting techniques in network emulation environments.

Contribution

It provides a comparative analysis of AI tools' ability to generate functional SDN controller code, highlighting strengths and limitations of each approach.

Findings

ChatGPT and DeepSeek produced more consistent, higher-quality code.

All models could generate functional controllers, but some required manual fixes.

Performance varied depending on prompting technique and tool used.

Abstract

Generative Artificial Intelligence (AI) tools have been used to generate human-like content across multiple domains (e.g., sound, image, text, and programming). However, their reliability in terms of correctness and functionality in novel contexts such as programmable networks remains unclear. Hence, this paper presents an empirical evaluation of the source code of a POX controller generated by different AI tools, namely ChatGPT, Copilot, DeepSeek, and BlackBox.ai. To evaluate such a code, three networking tasks of increasing complexity were defined and for each task, zero-shot and few-shot prompting techniques were input to the tools. Next, the output code was tested in emulated network topologies with Mininet and analyzed according to functionality, correctness, and the need for manual fixes. Results show that all evaluated models can produce functional controllers. However, ChatGPT and DeepSeek exhibited higher consistency and code quality, while Copilot and BlackBox.ai required more adjustments.