XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML

TL;DR

XAutoLM is a novel meta-learning and AutoML framework that significantly improves the efficiency and effectiveness of fine-tuning language models by reusing past experiences to optimize model selection and hyperparameters.

Contribution

It introduces a comprehensive automated framework that leverages meta-learning to enhance resource-efficient language model fine-tuning, outperforming existing methods.

Findings

Surpasses zero-shot optimizer’s peak F1 on five of six tasks.

Reduces mean evaluation time of pipelines by up to 4.5x.

Uncovers up to 50% more pipelines above the zero-shot Pareto front.

Abstract

Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimization, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and hyperparameter optimization (HPO) task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward valuable configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimizer's peak F1 on five of six tasks, cuts mean evaluation time of pipelines by up to 4.5x, reduces search error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyze resource-efficient, Green AI fine-tuning in the NLP community.