Positive-Congruent Training: Towards Regression-Free Model Updates

TL;DR

This paper introduces Positive-Congruent (PC) training with Focal Distillation to reduce negative prediction flips in model updates, maintaining accuracy while improving consistency with previous models.

Contribution

It proposes a novel PC training method that emphasizes positive predictions and leverages ensemble reference models to minimize negative flips without sacrificing accuracy.

Findings

Focal Distillation effectively reduces negative flips.

Ensemble reference models further decrease inconsistencies.

Method maintains high accuracy while improving model congruency.

Abstract

Reducing inconsistencies in the behavior of different versions of an AI system can be as important in practice as reducing its overall error. In image classification, sample-wise inconsistencies appear as "negative flips": A new model incorrectly predicts the output for a test sample that was correctly classified by the old (reference) model. Positive-congruent (PC) training aims at reducing error rate while at the same time reducing negative flips, thus maximizing congruency with the reference model only on positive predictions, unlike model distillation. We propose a simple approach for PC training, Focal Distillation, which enforces congruence with the reference model by giving more weights to samples that were correctly classified. We also found that, if the reference model itself can be chosen as an ensemble of multiple deep neural networks, negative flips can be further reduced without affecting the new model's accuracy.