Residual Stream Analysis of Overfitting And Structural Disruptions

TL;DR

This paper analyzes how safety fine-tuning causes overfitting in LLMs, leading to false refusals, and introduces a new regularizer, VCL, to mitigate this issue while preserving model performance.

Contribution

It introduces FlowLens for residual-stream analysis and proposes VCL, a novel regularizer that reduces overfitting and false refusals in safety fine-tuning of LLMs.

Findings

FlowLens reveals variance concentration causes false refusals.

VCL reduces false refusals by over 35 percentage points.

VCL maintains or improves benchmark performance.

Abstract

Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets, where unsafe prompts are paired with standard refusal templates, often leads to false refusals, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy and 2-gram diversity (0.048) compared to general instruction data. To uncover the root cause, we introduce FlowLens, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63 percent to 84 percent as safety data increases from 0 percent to 40 percent). Guided by these insights, we propose Variance Concentration Loss (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.