Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models

TL;DR

This paper introduces Portable Reward Tuning (PRT), a novel fine-tuning approach that trains a reward model to enable reusable fine-tuning across different foundation models, reducing inference overhead while maintaining accuracy.

Contribution

PRT reformulates fine-tuning as reward maximization, allowing a single reward model to be used with various foundation models without additional inference overhead.

Findings

PRT achieves comparable accuracy to inference-time tuning methods.

PRT reduces inference cost compared to traditional fine-tuning.

Effective across both vision and language models.

Abstract

While foundation models have been exploited for various expert tasks through fine-tuning, any foundation model will become outdated due to its old knowledge or limited capability. Thus the underlying foundation model should be eventually replaced by new ones, which leads to repeated cost of fine-tuning these new models. Existing work addresses this problem by inference-time tuning, i.e., modifying the output probabilities from the new foundation model with the outputs from the old foundation model and its fine-tuned model, which involves an additional overhead in inference by the latter two models. In this paper, we propose a new fine-tuning principle, Portable Reward Tuning (PRT), that reduces the inference overhead by its nature, based on the reformulation of fine-tuning as the reward maximization. Specifically, instead of fine-tuning parameters of the foundation models, PRT trains the reward model explicitly through the same loss function as in fine-tuning. During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels) through the formulation of reward maximization. Experimental results, covering both vision and language models, demonstrate that the PRT-trained model can achieve comparable accuracy to the existing work of inference-time tuning, with less inference cost.