Optimization-Inspired Few-Shot Adaptation for Large Language Models

TL;DR

This paper introduces OFA, a novel optimization-inspired method for few-shot adaptation of large language models that improves efficiency and performance without extra trainable parameters.

Contribution

It reinterprets LLM forward passes as optimization steps and proposes a parameterization that learns preconditioners to enhance few-shot adaptation.

Findings

OFA outperforms existing few-shot adaptation methods.

The method improves optimization efficiency and convergence.

OFA achieves superior results across various tasks.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via fine-tuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as in-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: in-context learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.