Mitigating Negative Flips via Margin Preserving Training

TL;DR

This paper introduces a margin-preserving training method that reduces negative flips in image classification models when updating with new classes, maintaining consistency and accuracy across model versions.

Contribution

It proposes a novel margin calibration approach combined with focal distillation to mitigate negative flips while preserving accuracy on both old and new classes.

Findings

Significantly reduces negative flip rate in experiments

Maintains high overall accuracy on benchmark datasets

Effective across various class-incremental scenarios

Abstract

Minimizing inconsistencies across successive versions of an AI system is as crucial as reducing the overall error. In image classification, such inconsistencies manifest as negative flips, where an updated model misclassifies test samples that were previously classified correctly. This issue becomes increasingly pronounced as the number of training classes grows over time, since adding new categories reduces the margin of each class and may introduce conflicting patterns that undermine their learning process, thereby degrading performance on the original subset. To mitigate negative flips, we propose a novel approach that preserves the margins of the original model while learning an improved one. Our method encourages a larger relative margin between the previously learned and newly introduced classes by introducing an explicit margin-calibration term on the logits. However, overly constraining the logit margin for the new classes can significantly degrade their accuracy compared to a new independently trained model. To address this, we integrate a double-source focal distillation loss with the previous model and a new independently trained model, learning an appropriate decision margin from both old and new data, even under a logit margin calibration. Extensive experiments on image classification benchmarks demonstrate that our approach consistently reduces the negative flip rate with high overall accuracy.