Unveiling the Basin-Like Loss Landscape in Large Language Models

TL;DR

This paper uncovers basin-like structures in the loss landscape of large language models, showing how model scale and fine-tuning influence stability, capabilities, and robustness, with implications for improving model training and safety.

Contribution

It introduces the concept of basins in LLM loss landscapes, linking model scale, fine-tuning, and robustness, and provides theoretical bounds on performance degradation.

Findings

Larger models have more expansive stability basins.

Benign fine-tuning within basins preserves capabilities.

Adversarial fine-tuning exploits worst-case directions, degrading performance.

Abstract

We discover the emergence of \textit{basins} in the loss landscape of large language models. As model scale increases, LLMs become progressively more resilient to random perturbations in the parameter space, giving rise to expansive stability regions where models exhibit nearly identical performance, but outside of which their capabilities collapse. We observe that pre-training creates a \textit{basic capability} basin, and subsequent alignment fine-tuning forms \textit{specific capability} basins (e.g., safety, math, coding). Thus, we argue that benign fine-tuning confined to the basin should preserve prior capabilities. Besides, we also analyze the loss landscape for worst-case directions, which is consistently sharp and detrimental. We find that adversarial fine-tuning moves along the nearly worst-case directions, thus rapidly degrading model capabilities. Finally, we provide a theoretical analysis demonstrating that the basin size bounds the performance degradation of any fine-tuning, including the adversarial ones, while also guaranteeing the model robustness w.r.t. input perturbations, suggesting the benefit of enlarging basins.