Differentiable Entailment for Parameter Efficient Few Shot Learning

TL;DR

This paper introduces a parameter-efficient method for few-shot learning that reformulates tasks as entailment problems and optimizes only a small subset of model parameters, enabling practical deployment with minimal performance tradeoff.

Contribution

It proposes a novel approach combining entailment reformulation and differentiable optimization of tokens, achieving competitive results by updating only 3% of parameters.

Findings

Optimizes only 3% of model parameters for few-shot learning.

Achieves competitive performance with efficient parameter updates.

Enables batched inference for practical deployment.

Abstract

Few-shot learning allows pre-trained language models to adapt to downstream tasks while using a limited number of training examples. However, practical applications are limited when all model parameters must be optimized. In this work we apply a new technique for parameter efficient few shot learning while adopting a strict definition of parameter efficiency. Our training method combines 1) intermediate training by reformulating natural language tasks as entailment tasks \cite{wang_entailment_2021} and 2) differentiable optimization of template and label tokens \cite{zhang_differentiable_2021}. We quantify the tradeoff between parameter efficiency and performance in the few-shot regime and propose a simple model agnostic approach that can be extended to any task By achieving competitive performance while only optimizing 3\% of a model's parameters and allowing for batched inference, we allow for more efficient practical deployment of models.