Automated Algorithm Design for Auto-Tuning Optimizers

TL;DR

This paper introduces a novel approach using large language models to automatically generate and refine optimization algorithms for auto-tuning high-performance applications, significantly improving performance across diverse tasks.

Contribution

It presents a framework that leverages LLMs to synthesize, test, and refine auto-tuning algorithms tailored to specific problems, outperforming existing human-designed methods.

Findings

Generated algorithms show an average 30.7% performance improvement with additional problem info.

LLM-generated optimizers can rival or outperform state-of-the-art algorithms.

Best algorithms achieve an average 72.4% improvement over existing optimizers.

Abstract

Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning.