Backbone Augmented Training for Adaptations

TL;DR

This paper introduces Backbone Augmented Training (BAT), a novel method that leverages pre-training backbone data to improve adaptation of large models, especially when adaptation data is scarce.

Contribution

The paper proposes BAT, formulates and proves key propositions validating its effectiveness, and introduces ALBAT, an algorithm for efficient adaptation in data-scarce scenarios.

Findings

BAT improves adaptation performance with limited data.

Theoretical propositions validate the effectiveness of BAT.

ALBAT algorithm enhances personalization and language generation tasks.

Abstract

Adaptations facilitate efficient training of large backbone models, including diffusion models for image generation and transformer-based language models. While various adaptation techniques enhance performance with minimal computational resources, limited adaptation data often leads to challenges in training. To address this, we focus on the enormous amount of backbone data used to pre-train the backbone models. We propose Backbone Augmented Training (BAT), a method that leverages backbone data to augment the adaptation dataset. First, we formulate and prove two mathematical key propositions: one establishes the validity of BAT, while the other identifies a condition under which BAT benefits adaptation. Furthermore, we introduce an advanced data selection scheme that satisfies these propositions and present ALBAT algorithm to implement this approach. ALBAT efficiently enhances adaptation training in both personalization and language generation tasks with scarce data.