Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights

TL;DR

The paper introduces Drag-and-Drop LLMs (DnD), a prompt-conditioned parameter generator that rapidly produces task-specific LoRA weights without training, significantly reducing overhead and improving performance on various benchmarks.

Contribution

DnD is a novel method that maps prompts directly to LoRA weights, eliminating the need for per-task training and enabling fast, zero-shot adaptation of large language models.

Findings

Up to 12,000× lower overhead than full fine-tuning

Up to 30% performance gains on unseen tasks

Robust cross-domain generalization without target data

Abstract

Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate optimization run for every downstream dataset. We introduce \textbf{Drag-and-Drop LLMs (\textit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to \textbf{12,000$\times$} lower overhead than full fine-tuning, ii) average gains up to \textbf{30\%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization despite never seeing the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs. Our project is available at \href{https://jerryliang24.github.io/DnD}{https://jerryliang24.github.io/DnD}.