A Timeline and Analysis for Representation Plasticity in Large Language Models

TL;DR

This paper investigates how large language models' internal representations, especially regarding 'honesty', change during fine-tuning, revealing a critical window of plasticity that can inform better alignment and intervention strategies.

Contribution

It provides the first detailed analysis of representation plasticity in LLMs during fine-tuning, highlighting a general pattern of a critical window for effective steering.

Findings

Early steering shows high plasticity in models.

Later stages exhibit a responsive critical window.

Pattern of plasticity is consistent across architectures.

Abstract

The ability to steer AI behavior is crucial to preventing its long term dangerous and catastrophic potential. Representation Engineering (RepE) has emerged as a novel, powerful method to steer internal model behaviors, such as "honesty", at a top-down level. Understanding the steering of representations should thus be placed at the forefront of alignment initiatives. Unfortunately, current efforts to understand plasticity at this level are highly neglected. This paper aims to bridge the knowledge gap and understand how LLM representation stability, specifically for the concept of "honesty", and model plasticity evolve by applying steering vectors extracted at different fine-tuning stages, revealing differing magnitudes of shifts in model behavior. The findings are pivotal, showing that while early steering exhibits high plasticity, later stages have a surprisingly responsive critical window. This pattern is observed across different model architectures, signaling that there is a general pattern of model plasticity that can be used for effective intervention. These insights greatly contribute to the field of AI transparency, addressing a pressing lack of efficiency limiting our ability to effectively steer model behavior.