TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs

TL;DR

This paper introduces TuCo, a method to quantify how fine-tuning influences individual responses of large language models by decomposing responses into pre-training and fine-tuning components, enabling detailed analysis of model behavior and safety.

Contribution

The paper presents a novel method for measuring the contribution of fine-tuning to individual LLM outputs using hidden state analysis and theoretical decomposition, advancing understanding of fine-tuning effects.

Findings

TuCo can steer model behavior by scaling fine-tuning components.

Attenuating fine-tuning effects reduces vulnerability to adversarial attacks.

TuCo correlates with safety and attack success in LLMs.

Abstract

Past work has studied the effects of fine-tuning on large language models' (LLMs) overall performance on certain tasks. However, a quantitative and systematic method for analyzing its effect on individual outputs is still lacking. Here, we propose a new method for measuring the contribution that fine-tuning makes to individual LLM responses, assuming access to the original pre-trained model. Our method tracks the model's intermediate hidden states, providing a more fine-grained insight into the effects of fine-tuning than a simple comparison of final outputs from pre-trained and fine-tuned models. We introduce and theoretically analyze an exact decomposition of any fine-tuned LLM into a pre-training component and a fine-tuning component. Empirically, we find that model behavior and performance can be steered by up- or down-scaling the fine-tuning component during the forward pass. Motivated by this finding and our theoretical analysis, we define the Tuning Contribution (TuCo) as the ratio of the magnitudes of the fine-tuning component to the pre-training component. We observe that three prominent adversarial attacks on LLMs circumvent safety measures in a way that reduces TuCo, and that TuCo is consistently lower on prompts where these attacks succeed compared to those where they do not. This suggests that attenuating the effect of fine-tuning on model outputs plays a role in the success of such attacks. In summary, TuCo enables the quantitative study of how fine-tuning influences model behavior and safety, and vice versa.