Large Language Models for IT Automation Tasks: Are We There Yet?

TL;DR

This paper assesses the ability of 14 open-source large language models to generate functional Ansible automation scripts for IT tasks, revealing significant limitations in state reasoning and domain-specific knowledge.

Contribution

Introduces ITAB, a new benchmark with 126 diverse IT automation tasks focusing on state reconciliation, and analyzes LLM failures in practical IT automation scenarios.

Findings

LLMs achieve pass@10 rates below 12% on ITAB.

Majority of errors stem from state reconciliation failures and module knowledge deficiencies.

Reliable IT automation requires advances in state reasoning and domain-specific understanding.

Abstract

LLMs show promise in code generation, yet their effectiveness for IT automation tasks, particularly for tools like Ansible, remains understudied. Existing benchmarks rely primarily on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools, such as Ansible. We present ITAB (IT Automation Task Benchmark), a benchmark of 126 diverse tasks (e.g., configuring servers, managing files) where each task accounts for state reconciliation: a property unique to IT automation tools. ITAB evaluates LLMs' ability to generate functional Ansible automation scripts via dynamic execution in controlled environments. We evaluate 14 open-source LLMs, none of which accomplish pass@10 at a rate beyond 12%. To explain these low scores, we analyze 1,411 execution failures across the evaluated LLMs and identify two main categories of prevalent semantic errors: failures in state reconciliation related reasoning (44.87% combined from variable (11.43%), host (11.84%), path(11.63%), and template (9.97%) issues) and deficiencies in module-specific execution knowledge (24.37% combined from Attribute and parameter (14.44%) and module (9.93%) errors). Our findings reveal key limitations in open-source LLMs' ability to track state changes and apply specialized module knowledge, indicating that reliable IT automation will require major advances in state reasoning and domain-specific execution understanding.