Can LLMs Configure Software Tools

TL;DR

This paper explores using Large-Language Models to improve software tool configuration by identifying initial conditions and narrowing search spaces, aiming to enhance efficiency and reproducibility in complex, high-dimensional settings.

Contribution

It introduces a novel approach leveraging LLMs like Chat-GPT for software configuration, addressing limitations of traditional methods and demonstrating potential benefits through experiments.

Findings

LLMs can identify effective starting conditions for configurations

Response caching and domain-specific keyword effects observed in LLM outputs

LLMs show promise in expediting hyperparameter optimization processes

Abstract

In software engineering, the meticulous configuration of software tools is crucial in ensuring optimal performance within intricate systems. However, the complexity inherent in selecting optimal configurations is exacerbated by the high-dimensional search spaces presented in modern applications. Conventional trial-and-error or intuition-driven methods are both inefficient and error-prone, impeding scalability and reproducibility. In this study, we embark on an exploration of leveraging Large-Language Models (LLMs) to streamline the software configuration process. We identify that the task of hyperparameter configuration for machine learning components within intelligent applications is particularly challenging due to the extensive search space and performance-critical nature. Existing methods, including Bayesian optimization, have limitations regarding initial setup, computational cost, and convergence efficiency. Our work presents a novel approach that employs LLMs, such as Chat-GPT, to identify starting conditions and narrow down the search space, improving configuration efficiency. We conducted a series of experiments to investigate the variability of LLM-generated responses, uncovering intriguing findings such as potential response caching and consistent behavior based on domain-specific keywords. Furthermore, our results from hyperparameter optimization experiments reveal the potential of LLMs in expediting initialization processes and optimizing configurations. While our initial insights are promising, they also indicate the need for further in-depth investigations and experiments in this domain.