Towards Automated Machine Learning Research

TL;DR

This paper proposes a top-down, LLM-driven approach to automating machine learning research by generating and validating novel components, aiming to accelerate innovation beyond traditional AutoML methods.

Contribution

It introduces a novel framework that leverages Large Language Models for component-level innovation in machine learning, moving away from predefined sets and enhancing hypothesis generation.

Findings

Framework successfully generates novel ML components.

Uses LLMs to propose components beyond predefined sets.

Incorporates reward models to prioritize promising hypotheses.

Abstract

This paper explores a top-down approach to automating incremental advances in machine learning research through component-level innovation, facilitated by Large Language Models (LLMs). Our framework systematically generates novel components, validates their feasibility, and evaluates their performance against existing baselines. A key distinction of this approach lies in how these novel components are generated. Unlike traditional AutoML and NAS methods, which often rely on a bottom-up combinatorial search over predefined, hardcoded base components, our method leverages the cross-domain knowledge embedded in LLMs to propose new components that may not be confined to any hard-coded predefined set. By incorporating a reward model to prioritize promising hypotheses, we aim to improve the efficiency of the hypothesis generation and evaluation process. We hope this approach offers a new avenue for exploration and contributes to the ongoing dialogue in the field.