Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training

TL;DR

This paper introduces LayerTracer, a diagnostic framework that interprets layer importance in LLMs, guiding efficient layer freezing strategies for cost-effective continued pre-training.

Contribution

It provides an interpretable method to identify critical layers, demonstrating that training shallow layers while freezing deep layers improves performance and resource efficiency.

Findings

Deep layers are critical for task execution and stable against updates.

Training shallow layers while freezing deep layers outperforms full fine-tuning.

Hybrid models with high-quality modules in deep layers preserve knowledge effectively.

Abstract

Selective layer-wise updates are essential for low-cost continued pre-training of Large Language Models (LLMs), yet determining which layers to freeze or train remains an empirical black-box problem due to the lack of interpretable guidance. To address this issue, we propose LayerTracer, an architecture-agnostic diagnostic framework that reveals the evolution patterns of layer-wise representations and stability by locating task execution positions and quantifying layer sensitivity. Analysis results reveal that deep layers act as critical regions for task execution and maintain high stability against disruptive updates. Guided by this finding, we conduct three controlled continued pre-training trials to compare diverse freeze-train strategies, demonstrating that training shallow layers while freezing deep layers consistently outperforms full-parameter fine-tuning and the opposite allocation on both C-Eval and CMMLU benchmarks. We further present a hybrid model case study, which validates that placing high-quality pre-trained modules in deep layers effectively preserves inherent knowledge of the model. This work delivers a low-cost and interpretable solution for resource-constrained teams, offering actionable guidance for layer-wise parameter allocation in continued pre-training and hybrid model construction.