Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates

TL;DR

This paper investigates regressions in NLP model updates, proposing methods to measure, reduce, and analyze them, with techniques like knowledge distillation and ensemble methods to improve model stability.

Contribution

It introduces a regression measurement approach, formulates regression reduction as a constrained optimization, and empirically evaluates ensemble and distillation techniques for regression mitigation.

Findings

Regression is prevalent across NLP tasks in GLUE.

Knowledge distillation can approximately reduce regressions.

Ensemble methods further decrease regression errors.

Abstract

Behavior of deep neural networks can be inconsistent between different versions. Regressions during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regression measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimization problem, and further reduce it into a relaxed form which can be approximately optimized through knowledge distillation training method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CheckList behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods.