Tracking Universal Features Through Fine-Tuning and Model Merging

TL;DR

This paper investigates how features in a Transformer language model evolve, disappear, or remain stable when fine-tuned on different text domains and then merged, providing insights into feature stability in transfer learning.

Contribution

It introduces a method to analyze feature dynamics across fine-tuning and merging in small-scale models, revealing patterns of feature stability and transformation.

Findings

Features can be stable or change significantly across domains.

Model merging via spherical interpolation preserves some features while altering others.

Insights into feature behavior aid understanding of transfer learning processes.

Abstract

We study how features emerge, disappear, and persist across models fine-tuned on different domains of text. More specifically, we start from a base one-layer Transformer language model that is trained on a combination of the BabyLM corpus, and a collection of Python code from The Stack. This base model is adapted to two new domains of text: TinyStories, and the Lua programming language, respectively; and then these two models are merged using these two models using spherical linear interpolation. Our exploration aims to provide deeper insights into the stability and transformation of features across typical transfer-learning scenarios using small-scale models and sparse auto-encoders.