Visualizing and Understanding the Effectiveness of BERT

TL;DR

This paper visualizes BERT's loss landscapes and optimization paths to understand why pre-training enhances performance, revealing that pre-training leads to wider optima, robustness to overfitting, and layer invariance during fine-tuning.

Contribution

It introduces visualization techniques to analyze BERT's fine-tuning process, uncovering the reasons behind its effectiveness and the nature of its optimization landscape.

Findings

Pre-training results in wider, flatter optima facilitating easier optimization.

Fine-tuning BERT is robust to overfitting despite over-parameterization.

Lower layers are more invariant and learn more transferable representations.

Abstract

Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks. However, it is unclear why the pre-training-then-fine-tuning paradigm can improve performance and generalization capability across different tasks. In this paper, we propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. First, we find that pre-training reaches a good initial point across downstream tasks, which leads to wider optima and easier optimization compared with training from scratch. We also demonstrate that the fine-tuning procedure is robust to overfitting, even though BERT is highly over-parameterized for downstream tasks. Second, the visualization results indicate that fine-tuning BERT tends to generalize better because of the flat and wide optima, and the consistency between the training loss surface and the generalization error surface. Third, the lower layers of BERT are more invariant during fine-tuning, which suggests that the layers that are close to input learn more transferable representations of language.