LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

TL;DR

LLaMA-Adapter introduces a lightweight, efficient fine-tuning method for LLaMA that uses zero-initialized attention and minimal parameters, achieving high-quality instruction-following and multi-modal performance with low computational cost.

Contribution

The paper proposes a novel zero-initialized attention mechanism and a lightweight adaptation approach for efficient fine-tuning of large language models like LLaMA.

Findings

Achieves comparable performance to fully fine-tuned models with only 1.2M additional parameters.

Extends to multi-modal instruction learning, outperforming existing methods on benchmarks.

Demonstrates the generalization of zero-initialized attention to vision and language tasks.

Abstract

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the word tokens at higher transformer layers. Then, a zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Besides language commands, our approach can be simply extended to multi-modal instructions for learning image-conditioned LLaMA model, which achieves superior reasoning performance on ScienceQA and COCO Caption benchmarks. Furthermore, we also evaluate the zero-initialized attention mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on traditional vision and language tasks, demonstrating the superior generalization capacity of our approach. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.