Large Language Models Synergize with Automated Machine Learning

TL;DR

This paper introduces a fully automated approach combining large language models and autoML to generate, optimize, and evaluate complete machine learning workflows from textual descriptions, outperforming existing methods in most tested tasks.

Contribution

It presents a novel method that automates the generation and optimization of ML programs by breaking them into parts, ensuring compatibility, and using autoML for evaluation, advancing program synthesis for ML tasks.

Findings

Outperforms existing methods in 10 out of 12 ML tasks

Automates the entire ML workflow from data to post-processing

AutoML enhances the performance of generated ML programs

Abstract

Recently, program synthesis driven by large language models (LLMs) has become increasingly popular. However, program synthesis for machine learning (ML) tasks still poses significant challenges. This paper explores a novel form of program synthesis, targeting ML programs, by combining LLMs and automated machine learning (autoML). Specifically, our goal is to fully automate the generation and optimization of the code of the entire ML workflow, from data preparation to modeling and post-processing, utilizing only textual descriptions of the ML tasks. To manage the length and diversity of ML programs, we propose to break each ML program into smaller, manageable parts. Each part is generated separately by the LLM, with careful consideration of their compatibilities. To ensure compatibilities, we design a testing technique for ML programs. Unlike traditional program synthesis, which typically relies on binary evaluations (i.e., correct or incorrect), evaluating ML programs necessitates more than just binary judgments. Our approach automates the numerical evaluation and optimization of these programs, selecting the best candidates through autoML techniques. In experiments across various ML tasks, our method outperforms existing methods in 10 out of 12 tasks for generating ML programs. In addition, autoML significantly improves the performance of the generated ML programs. In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process. The implementation of our method is available at https://github.com/JLX0/llm-automl.