AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks

TL;DR

AutoMMLab introduces an end-to-end AutoML system that converts natural language requests into deployable computer vision models, featuring a new benchmark and a novel hyperparameter optimization method using large language models.

Contribution

The paper presents AutoMMLab, a comprehensive AutoML platform for computer vision, and proposes HPO-LLaMA, a new LLM-based hyperparameter optimization algorithm, advancing automation and efficiency.

Findings

HPO-LLaMA significantly improves hyperparameter tuning efficiency.

AutoMMLab enables non-experts to build models via natural language.

LAMP benchmark facilitates evaluation of end-to-end AutoML components.

Abstract

Automated machine learning (AutoML) is a collection of techniques designed to automate the machine learning development process. While traditional AutoML approaches have been successfully applied in several critical steps of model development (e.g. hyperparameter optimization), there lacks a AutoML system that automates the entire end-to-end model production workflow for computer vision. To fill this blank, we propose a novel request-to-model task, which involves understanding the user's natural language request and execute the entire workflow to output production-ready models. This empowers non-expert individuals to easily build task-specific models via a user-friendly language interface. To facilitate development and evaluation, we develop a new experimental platform called AutoMMLab and a new benchmark called LAMP for studying key components in the end-to-end request-to-model pipeline. Hyperparameter optimization (HPO) is one of the most important components for AutoML. Traditional approaches mostly rely on trial-and-error, leading to inefficient parameter search. To solve this problem, we propose a novel LLM-based HPO algorithm, called HPO-LLaMA. Equipped with extensive knowledge and experience in model hyperparameter tuning, HPO-LLaMA achieves significant improvement of HPO efficiency. Dataset and code are available at https://github.com/yang-ze-kang/AutoMMLab.