Sustainability Analysis of Prompt Strategies for SLM-based Automated Test Generation

TL;DR

This study systematically evaluates how different prompt engineering strategies affect the environmental sustainability and effectiveness of automated test generation using Small Language Models, highlighting trade-offs between test quality and resource consumption.

Contribution

First comprehensive analysis of prompt engineering's impact on sustainability in SLM-based automated testing, considering multiple environmental and performance metrics.

Findings

Prompt strategies significantly influence sustainability outcomes independently of model choice.

Reasoning strategies like Chain of Thought improve coverage but increase resource consumption.

Simpler strategies like Zero-Shot offer a good balance of coverage and environmental impact.

Abstract

The growing adoption of prompt-based automation in software testing raises important issues regarding its computational and environmental sustainability. Existing sustainability studies in AI-driven testing primarily focus on large language models, leaving the impact of prompt engineering strategies largely unexplored - particularly in the context of Small Language Models (SLMs). This gap is critical, as prompt design directly influences inference behavior, execution cost, and resource utilization, even when model size is fixed. To the best of our knowledge, this paper presents the first systematic sustainability evaluation of prompt engineering strategies for automated test generation using SLMs. We analyze seven prompt strategies across three open-source SLMs under a controlled experimental setup. Our evaluation jointly considers execution time, token usage, energy consumption, carbon emissions, and coverage test quality, the latter assessed through coverage analysis of the generated test scripts. The results show that prompt strategies have a substantial and independent impact on sustainability outcomes, often outweighing the effect of model choice. Reasoning intensive strategies such as Chain of Thought and Self-Consistency achieve higher coverage but incur significantly higher execution time, energy consumption, and carbon emissions. In contrast, simpler strategies such as Zero-Shot and ReAct deliver competitive coverage test quality with markedly lower environmental cost, while Least-to-Most and Program of Thought offer balanced trade-offs.