Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference

TL;DR

Conditional Adapter (CoDA) is a transfer learning method that significantly enhances inference speed and parameter efficiency across multiple modalities by using conditional computation with minimal accuracy loss.

Contribution

CoDA introduces a novel conditional computation approach that generalizes adapter methods, enabling faster inference with minimal accuracy trade-offs.

Findings

Achieves 2x to 8x inference speed-up over state-of-the-art adapters

Maintains comparable accuracy with moderate to no loss

Effective across language, vision, and speech tasks

Abstract

We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.