Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models

TL;DR

This paper investigates how fine-tuning can recover BERT's performance after parameter corruption, revealing that corruption at lower layers causes more damage and that recovery is limited, informing robustness strategies.

Contribution

It introduces a systematic study of BERT's robustness to parameter corruption and how fine-tuning recovers performance, highlighting the importance of layer-specific effects.

Findings

Corrupted models struggle to fully recover original performance.

Lower-layer corruption is more damaging than upper-layer corruption.

Fine-tuning offers limited recovery after significant corruption.

Abstract

Language models like BERT excel at sentence classification tasks due to extensive pre-training on general data, but their robustness to parameter corruption is unexplored. To understand this better, we look at what happens if a language model is "broken", in the sense that some of its parameters are corrupted and then recovered by fine-tuning. Strategically corrupting BERT variants at different levels, we find corrupted models struggle to fully recover their original performance, with higher corruption causing more severe degradation. Notably, bottom-layer corruption affecting fundamental linguistic features is more detrimental than top-layer corruption. Our insights contribute to understanding language model robustness and adaptability under adverse conditions, informing strategies for developing resilient NLP systems against parameter perturbations.