Provable Meta-Learning with Low-Rank Adaptations

TL;DR

This paper introduces a provable meta-learning framework using low-rank adaptations that guarantees better adaptability for unseen tasks, supported by theoretical analysis and empirical validation on vision and language data.

Contribution

It provides the first theoretical guarantees for PEFT-based meta-learning with low-rank adaptations and demonstrates its practical benefits over standard retraining methods.

Findings

Provably suboptimal performance of standard retraining for linear models.

Theoretical guarantees for the proposed meta-learning method.

Empirical improvements on synthetic, vision, and language tasks.

Abstract

The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained models require additional training stages to become effective for downstream applications. In the multi-task setting, prior works have shown empirically that specific meta-learning approaches for preparing a model for future adaptation through parameter-efficient fine-tuning (PEFT) can outperform standard retraining methods, but the mechanism of the benefits of meta-learning has been largely unexplored. We introduce a framework for generic PEFT-based meta-learning to learn a model that can easily adapt to unseen tasks. For linear models using LoRA, we show that standard retraining is provably suboptimal for finding an adaptable set of parameters and provide strict performance guarantees for our proposed method. We verify these theoretical insights through experiments on synthetic data as well as real-data vision and language tasks. We observe significant performance benefits using a simple implementation of our proposed meta-learning scheme during retraining relative to the conventional approach.