Text-to-LoRA: Instant Transformer Adaption

TL;DR

Text-to-LoRA (T2L) is a hypernetwork that rapidly adapts large language models to new tasks using natural language descriptions, eliminating the need for extensive fine-tuning and enabling zero-shot generalization.

Contribution

We introduce T2L, a hypernetwork that constructs LoRA adapters from natural language descriptions, allowing fast, task-specific adaptation of LLMs without costly fine-tuning.

Findings

T2L matches the performance of task-specific adapters.

It can compress and generalize across multiple tasks.

Zero-shot generalization to unseen tasks is achieved.

Abstract

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora