\`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting

TL;DR

This paper presents e0-la-carte Prompt Tuning (APT), a method for training modular prompts on separate data sources that can be combined at inference to create customized models without retraining.

Contribution

The paper introduces e0-la-carte Prompt Tuning, enabling modular prompt composition from separate data sources for flexible, efficient model customization.

Findings

Models built with e0-la-carte prompts achieve within 5% accuracy of combined-source models.

e0-la-carte prompts enable flexible model assembly without retraining.

State-of-the-art performance on Split CIFAR-100 and CORe50 benchmarks.

Abstract

We introduce \`A-la-carte Prompt Tuning (APT), a transformer-based scheme to tune prompts on distinct data so that they can be arbitrarily composed at inference time. The individual prompts can be trained in isolation, possibly on different devices, at different times, and on different distributions or domains. Furthermore each prompt only contains information about the subset of data it was exposed to during training. During inference, models can be assembled based on arbitrary selections of data sources, which we call "\`a-la-carte learning". \`A-la-carte learning enables constructing bespoke models specific to each user's individual access rights and preferences. We can add or remove information from the model by simply adding or removing the corresponding prompts without retraining from scratch. We demonstrate that \`a-la-carte built models achieve accuracy within $5\%$ of models trained on the union of the respective sources, with comparable cost in terms of training and inference time. For the continual learning benchmarks Split CIFAR-100 and CORe50, we achieve state-of-the-art performance.